diff options
148 files changed, 4525 insertions, 1565 deletions
diff --git a/INSTALL.md b/INSTALL.md index 1061c5187a..f0fdaf3982 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -424,7 +424,7 @@ as before, but the build process will take a much longer time. ### Building in Git ### When building in a Git working directory you also have to have a GNU `autoconf` -of at least version 2.59 on your system. This since you need to generate the +of at least version 2.59 on your system, because you need to generate the `configure` scripts before you can start building. The `configure` scripts are generated by invoking `./otp_build autoconf` in @@ -436,7 +436,7 @@ when checking out a branch. Regenerated `configure` scripts imply that you have to run `configure` and build again. > *NOTE*: Running `./otp_build autoconf` is **not** needed when building -> an unmodified version the released source. +> an unmodified version of the released source. Other useful information can be found at our github wiki: <http://wiki.github.com/erlang/otp> diff --git a/erts/doc/src/erts_alloc.xml b/erts/doc/src/erts_alloc.xml index 51a4a2bca0..90347824d5 100644 --- a/erts/doc/src/erts_alloc.xml +++ b/erts/doc/src/erts_alloc.xml @@ -78,14 +78,20 @@ segments are allocated, cached segments are used if possible instead of creating new segments. This in order to reduce the number of system calls made.</item> + <tag><c>sbmbc_alloc</c></tag> + <item>Allocator used by other allocators for allocation of carriers + where only small blocks are placed. Currently this allocator is + disabled by default.</item> </taglist> <p><c>sys_alloc</c> and <c>fix_alloc</c> are always enabled and cannot be disabled. <c>mseg_alloc</c> is always enabled if it is available and an allocator that uses it is enabled. All other allocators can be <seealso marker="#M_e">enabled or disabled</seealso>. By default all allocators are enabled. - When an allocator is disabled, <c>sys_alloc</c> - is used instead of the disabled allocator.</p> + When an allocator is disabled, <c>sys_alloc</c> is used instead of + the disabled allocator. <c>sbmbc_alloc</c> is an exception. If + <c>sbmbc_alloc</c> is disabled, other allocators will not handle + small blocks in separate carriers.</p> <p>The main idea with the <c>erts_alloc</c> library is to separate memory blocks that are used differently into different memory areas, and by this achieving less memory fragmentation. By @@ -103,15 +109,20 @@ following does <em>not</em> apply to them.</p> <p>An allocator manages multiple areas, called carriers, in which memory blocks are placed. A carrier is either placed in a - separate memory segment (allocated via <c>mseg_alloc</c>) or in - the heap segment (allocated via <c>sys_alloc</c>). Multiblock + separate memory segment (allocated via <c>mseg_alloc</c>), in + the heap segment (allocated via <c>sys_alloc</c>), or inside + another carrier (in case it is a carrier created by + <c>sbmbc_alloc</c>). Multiblock carriers are used for storage of several blocks. Singleblock carriers are used for storage of one block. Blocks that are larger than the value of the singleblock carrier threshold (<seealso marker="#M_sbct">sbct</seealso>) parameter are placed - in singleblock carriers. Blocks smaller than the value of the - <c>sbct</c> parameter are placed in multiblock - carriers. Normally an allocator creates a "main multiblock + in singleblock carriers. Blocks that are smaller than the value + of the <c>sbct</c> parameter are placed in multiblock + carriers. Blocks that are smaller than the small block multiblock + carrier threshold (<seealso marker="#M_sbmbct">sbmbct</seealso>) + will be placed in multiblock carriers only used for small blocks. + Normally an allocator creates a "main multiblock carrier". Main multiblock carriers are never deallocated. The size of the main multiblock carrier is determined by the value of the <seealso marker="#M_mmbcs">mmbcs</seealso> parameter.</p> @@ -133,8 +144,11 @@ <c>sbct</c> parameter should be larger than the value of the <c>lmbcs</c> parameter, the allocator may have to create multiblock carriers that are larger than the value of the - <c>lmbcs</c> parameter, though. Singleblock carriers allocated - via <c>mseg_alloc</c> are sized to whole pages.</p> + <c>lmbcs</c> parameter, though. The size of multiblock carriers + for small blocks is determined by the small block multiblock + carrier size (<seealso marker="#M_sbmbcs">sbmbcs</seealso>). + Singleblock carriers allocated via <c>mseg_alloc</c> are sized + to whole pages.</p> <p>Sizes of carriers allocated via <c>sys_alloc</c> are decided based on the value of the <c>sys_alloc</c> carrier size (<seealso marker="#Muycs">ycs</seealso>) parameter. The size of @@ -166,6 +180,14 @@ used. The time complexity is proportional to log N, where N is the number of free blocks.</p> </item> + <tag>Address order first fit</tag> + <item> + <p>Strategy: Find the block with the lowest address that satisfies the + requested block size.</p> + <p>Implementation: A balanced binary search tree is + used. The time complexity is proportional to log N, where + N is the number of free blocks.</p> + </item> <tag>Good fit</tag> <item> <p>Strategy: Try to find the best fit, but settle for the best fit @@ -194,6 +216,11 @@ </taglist> </section> + <note><p> + Currently only allocators using the best fit and the address order + best fit strategies are able to use "small block multi block carriers". + </p></note> + <section> <marker id="flags"></marker> <title>System Flags Effecting erts_alloc</title> @@ -215,6 +242,7 @@ the currently present allocators:</p> <list type="bulleted"> <item><c>B: binary_alloc</c></item> + <item><c>C: sbmbc_alloc</c></item> <item><c>D: std_alloc</c></item> <item><c>E: ets_alloc</c></item> <item><c>F: fix_alloc</c></item> @@ -300,11 +328,11 @@ subsystem identifier, only the specific allocator identified will be effected:</p> <taglist> - <tag><marker id="M_as"><c><![CDATA[+M<S>as bf|aobf|gf|af]]></c></marker></tag> + <tag><marker id="M_as"><c><![CDATA[+M<S>as bf|aobf|aoff|gf|af]]></c></marker></tag> <item> Allocation strategy. Valid strategies are <c>bf</c> (best fit), - <c>aobf</c> (address order best fit), <c>gf</c> (good fit), - and <c>af</c> (a fit). See + <c>aobf</c> (address order best fit), <c>aoff</c> (address order first fit), + <c>gf</c> (good fit), and <c>af</c> (a fit). See <seealso marker="#strategy">the description of allocation strategies</seealso> in "the <c>alloc_util</c> framework" section.</item> <tag><marker id="M_asbcst"><c><![CDATA[+M<S>asbcst <size>]]></c></marker></tag> <item> @@ -395,6 +423,20 @@ threshold will be placed in singleblock carriers. Blocks smaller than this threshold will be placed in multiblock carriers.</item> + <tag><marker id="M_sbmbcs"><c><![CDATA[+M<S>sbmbcs <size>]]></c></marker></tag> + <item> + Small block multiblock carrier size (in bytes). Memory blocks smaller + than the small block multiblock carrier threshold + (<seealso marker="#M_sbmbct">sbmbct</seealso>) will be placed in + multiblock carriers used for small blocks only. This parameter + determines the size of such carriers. + </item> + <tag><marker id="M_sbmbct"><c><![CDATA[+M<S>sbmbct <size>]]></c></marker></tag> + <item> + Small block multiblock carrier threshold (in bytes). Memory blocks + smaller than this threshold will be placed in multiblock carriers + used for small blocks only. + </item> <tag><marker id="M_smbcs"><c><![CDATA[+M<S>smbcs <size>]]></c></marker></tag> <item> Smallest (<c>mseg_alloc</c>) multiblock carrier size (in diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index d9362a2a8f..b658e79378 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -742,7 +742,7 @@ RUN_OBJS = \ $(OBJDIR)/erl_bif_re.o $(OBJDIR)/erl_unicode.o \ $(OBJDIR)/packet_parser.o $(OBJDIR)/safe_hash.o \ $(OBJDIR)/erl_zlib.o $(OBJDIR)/erl_nif.o \ - $(OBJDIR)/erl_bif_binary.o + $(OBJDIR)/erl_bif_binary.o $(OBJDIR)/erl_ao_firstfit_alloc.o ifeq ($(TARGET),win32) DRV_OBJS = \ diff --git a/erts/emulator/beam/beam_emu.c b/erts/emulator/beam/beam_emu.c index fb90a7d4f7..937b3d9e53 100644 --- a/erts/emulator/beam/beam_emu.c +++ b/erts/emulator/beam/beam_emu.c @@ -3561,7 +3561,7 @@ void process_main(void) * Operands: NotUsed Live Dst */ do_bs_init_bits_known: - num_bytes = (num_bits+7) >> 3; + num_bytes = ((Uint64)num_bits+(Uint64)7) >> 3; if (num_bits & 7) { alloc += ERL_SUB_BIN_SIZE; } diff --git a/erts/emulator/beam/beam_load.c b/erts/emulator/beam/beam_load.c index 57fe25453d..eb10ae59a8 100644 --- a/erts/emulator/beam/beam_load.c +++ b/erts/emulator/beam/beam_load.c @@ -332,20 +332,22 @@ typedef struct { Eterm* func_tab[1]; /* Pointers to each function. */ } LoadedCode; -#define GetTagAndValue(Stp, Tag, Val) \ - do { \ - BeamInstr __w; \ - GetByte(Stp, __w); \ - Tag = __w & 0x07; \ - if ((__w & 0x08) == 0) { \ - Val = __w >> 4; \ - } else if ((__w & 0x10) == 0) { \ - Val = ((__w >> 5) << 8); \ - GetByte(Stp, __w); \ - Val |= __w; \ - } else { \ - if (!get_int_val(Stp, __w, &(Val))) goto load_error; \ - } \ +#define GetTagAndValue(Stp, Tag, Val) \ + do { \ + BeamInstr __w; \ + GetByte(Stp, __w); \ + Tag = __w & 0x07; \ + if ((__w & 0x08) == 0) { \ + Val = __w >> 4; \ + } else if ((__w & 0x10) == 0) { \ + Val = ((__w >> 5) << 8); \ + GetByte(Stp, __w); \ + Val |= __w; \ + } else { \ + int __res = get_tag_and_value(Stp, __w, (Tag), &(Val)); \ + if (__res < 0) goto load_error; \ + Tag = (unsigned) __res; \ + } \ } while (0) @@ -489,8 +491,8 @@ static void load_printf(int line, LoaderState* context, char *fmt, ...); static int transform_engine(LoaderState* st); static void id_to_string(Uint id, char* s); static void new_genop(LoaderState* stp); -static int get_int_val(LoaderState* stp, Uint len_code, BeamInstr* result); -static int get_erlang_integer(LoaderState* stp, Uint len_code, BeamInstr* result); +static int get_tag_and_value(LoaderState* stp, Uint len_code, + unsigned tag, BeamInstr* result); static int new_label(LoaderState* stp); static void new_literal_patch(LoaderState* stp, int pos); static void new_string_patch(LoaderState* stp, int pos); @@ -1470,46 +1472,15 @@ load_code(LoaderState* stp) last_op->arity = 0; ASSERT(arity <= MAX_OPARGS); -#define GetValue(Stp, First, Val) \ - do { \ - if (((First) & 0x08) == 0) { \ - Val = (First) >> 4; \ - } else if (((First) & 0x10) == 0) { \ - BeamInstr __w; \ - GetByte(Stp, __w); \ - Val = (((First) >> 5) << 8) | __w; \ - } else { \ - if (!get_int_val(Stp, (First), &(Val))) goto load_error; \ - } \ - } while (0) - for (arg = 0; arg < arity; arg++) { - BeamInstr first; - - GetByte(stp, first); - last_op->a[arg].type = first & 0x07; + GetTagAndValue(stp, last_op->a[arg].type, last_op->a[arg].val); switch (last_op->a[arg].type) { case TAG_i: - if ((first & 0x08) == 0) { - last_op->a[arg].val = first >> 4; - } else if ((first & 0x10) == 0) { - BeamInstr w; - GetByte(stp, w); - ASSERT(first < 0x800); - last_op->a[arg].val = ((first >> 5) << 8) | w; - } else { - int i = get_erlang_integer(stp, first, &(last_op->a[arg].val)); - if (i < 0) { - goto load_error; - } - last_op->a[arg].type = i; - } - break; case TAG_u: - GetValue(stp, first, last_op->a[arg].val); + case TAG_q: + case TAG_o: break; case TAG_x: - GetValue(stp, first, last_op->a[arg].val); if (last_op->a[arg].val == 0) { last_op->a[arg].type = TAG_r; } else if (last_op->a[arg].val >= MAX_REG) { @@ -1518,7 +1489,6 @@ load_code(LoaderState* stp) } break; case TAG_y: - GetValue(stp, first, last_op->a[arg].val); if (last_op->a[arg].val >= MAX_REG) { LoadError1(stp, "invalid y register number: %u", last_op->a[arg].val); @@ -1526,7 +1496,6 @@ load_code(LoaderState* stp) last_op->a[arg].val += CP_SIZE; break; case TAG_a: - GetValue(stp, first, last_op->a[arg].val); if (last_op->a[arg].val == 0) { last_op->a[arg].type = TAG_n; } else if (last_op->a[arg].val >= stp->num_atoms) { @@ -1536,7 +1505,6 @@ load_code(LoaderState* stp) } break; case TAG_f: - GetValue(stp, first, last_op->a[arg].val); if (last_op->a[arg].val == 0) { last_op->a[arg].type = TAG_p; } else if (last_op->a[arg].val >= stp->num_labels) { @@ -1544,7 +1512,6 @@ load_code(LoaderState* stp) } break; case TAG_h: - GetValue(stp, first, last_op->a[arg].val); if (last_op->a[arg].val > 65535) { LoadError1(stp, "invalid range for character data type: %u", last_op->a[arg].val); @@ -1552,11 +1519,9 @@ load_code(LoaderState* stp) break; case TAG_z: { - BeamInstr ext_tag; unsigned tag; - GetValue(stp, first, ext_tag); - switch (ext_tag) { + switch (last_op->a[arg].val) { case 0: /* Floating point number */ { Eterm* hp; @@ -1648,7 +1613,8 @@ load_code(LoaderState* stp) break; } default: - LoadError1(stp, "invalid extended tag %d", ext_tag); + LoadError1(stp, "invalid extended tag %d", + last_op->a[arg].val); break; } } @@ -1659,7 +1625,6 @@ load_code(LoaderState* stp) } last_op->arity++; } -#undef GetValue ASSERT(arity == last_op->arity); @@ -2562,13 +2527,8 @@ should_gen_heap_bin(LoaderState* stp, GenOpArg Src) static int binary_too_big(LoaderState* stp, GenOpArg Size) { - return Size.type == TAG_u && ((Size.val >> (8*sizeof(Uint)-3)) != 0); -} - -static int -binary_too_big_bits(LoaderState* stp, GenOpArg Size) -{ - return Size.type == TAG_u && (((Size.val+7)/8) >> (8*sizeof(Uint)-3) != 0); + return Size.type == TAG_o || + (Size.type == TAG_u && ((Size.val >> (8*sizeof(Uint)-3)) != 0)); } static GenOp* @@ -4317,41 +4277,9 @@ load_printf(int line, LoaderState* context, char *fmt,...) erts_send_error_to_logger(context->group_leader, dsbufp); } - -static int -get_int_val(LoaderState* stp, Uint len_code, BeamInstr* result) -{ - Uint count; - Uint val; - - len_code >>= 5; - ASSERT(len_code < 8); - if (len_code == 7) { - LoadError0(stp, "can't load integers bigger than 8 bytes yet\n"); - } - count = len_code + 2; - if (count == 5) { - Uint msb; - GetByte(stp, msb); - if (msb == 0) { - count--; - } - GetInt(stp, 4, *result); - } else if (count <= 4) { - GetInt(stp, count, val); - *result = ((val << 8*(sizeof(val)-count)) >> 8*(sizeof(val)-count)); - } else { - LoadError1(stp, "too big integer; %d bytes\n", count); - } - return 1; - - load_error: - return 0; -} - - static int -get_erlang_integer(LoaderState* stp, Uint len_code, BeamInstr* result) +get_tag_and_value(LoaderState* stp, Uint len_code, + unsigned tag, BeamInstr* result) { Uint count; Sint val; @@ -4371,17 +4299,62 @@ get_erlang_integer(LoaderState* stp, Uint len_code, BeamInstr* result) if (len_code < 7) { count = len_code + 2; } else { - Uint tag; + unsigned sztag; UWord len_word; ASSERT(len_code == 7); - GetTagAndValue(stp, tag, len_word); - VerifyTag(stp, TAG_u, tag); + GetTagAndValue(stp, sztag, len_word); + VerifyTag(stp, sztag, TAG_u); count = len_word + 9; } /* - * Handle values up to the size of an int, meaning either a small or bignum. + * The value for tags except TAG_i must be an unsigned integer + * fitting in an Uint. If it does not fit, we'll indicate overflow + * by changing the tag to TAG_o. + */ + + if (tag != TAG_i) { + if (count == sizeof(Uint)+1) { + Uint msb; + + /* + * The encoded value has one more byte than an Uint. + * It will still fit in an Uint if the most significant + * byte is 0. + */ + GetByte(stp, msb); + GetInt(stp, sizeof(Uint), *result); + if (msb != 0) { + /* Overflow: Negative or too big. */ + return TAG_o; + } + } else if (count == sizeof(Uint)) { + /* + * The value must be positive (or the encoded value would + * have been one byte longer). + */ + GetInt(stp, count, *result); + } else if (count < sizeof(Uint)) { + GetInt(stp, count, *result); + + /* + * If the sign bit is set, the value is negative + * (not allowed). + */ + if (*result & ((Uint)1 << (count*8-1))) { + return TAG_o; + } + } else { + GetInt(stp, count, *result); + return TAG_o; + } + return tag; + } + + /* + * TAG_i: First handle values up to the size of an Uint (i.e. either + * a small or a bignum). */ if (count <= sizeof(val)) { diff --git a/erts/emulator/beam/erl_afit_alloc.c b/erts/emulator/beam/erl_afit_alloc.c index e8b594bb47..d397cd8848 100644 --- a/erts/emulator/beam/erl_afit_alloc.c +++ b/erts/emulator/beam/erl_afit_alloc.c @@ -43,9 +43,9 @@ /* Prototypes of callback functions */ static Block_t * get_free_block (Allctr_t *, Uint, - Block_t *, Uint); -static void link_free_block (Allctr_t *, Block_t *); -static void unlink_free_block (Allctr_t *, Block_t *); + Block_t *, Uint, Uint32); +static void link_free_block (Allctr_t *, Block_t *, Uint32); +static void unlink_free_block (Allctr_t *, Block_t *, Uint32); static Eterm info_options (Allctr_t *, char *, int *, @@ -72,6 +72,8 @@ erts_afalc_start(AFAllctr_t *afallctr, is a struct). */ Allctr_t *allctr = (Allctr_t *) afallctr; + init->sbmbct = 0; /* Small mbc not supported by afit */ + sys_memcpy((void *) afallctr, (void *) &nulled_state, sizeof(AFAllctr_t)); allctr->mbc_header_size = sizeof(Carrier_t); @@ -105,7 +107,8 @@ erts_afalc_start(AFAllctr_t *afallctr, } static Block_t * -get_free_block(Allctr_t *allctr, Uint size, Block_t *cand_blk, Uint cand_size) +get_free_block(Allctr_t *allctr, Uint size, Block_t *cand_blk, Uint cand_size, + Uint32 flags) { AFAllctr_t *afallctr = (AFAllctr_t *) allctr; @@ -123,7 +126,7 @@ get_free_block(Allctr_t *allctr, Uint size, Block_t *cand_blk, Uint cand_size) } static void -link_free_block(Allctr_t *allctr, Block_t *block) +link_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { AFFreeBlock_t *blk = (AFFreeBlock_t *) block; AFAllctr_t *afallctr = (AFAllctr_t *) allctr; @@ -144,7 +147,7 @@ link_free_block(Allctr_t *allctr, Block_t *block) } static void -unlink_free_block(Allctr_t *allctr, Block_t *block) +unlink_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { AFFreeBlock_t *blk = (AFFreeBlock_t *) block; AFAllctr_t *afallctr = (AFAllctr_t *) allctr; diff --git a/erts/emulator/beam/erl_alloc.c b/erts/emulator/beam/erl_alloc.c index cda404af5e..bbc8a445a7 100644 --- a/erts/emulator/beam/erl_alloc.c +++ b/erts/emulator/beam/erl_alloc.c @@ -50,6 +50,9 @@ #include "erl_bestfit_alloc.h" #define GET_ERL_AF_ALLOC_IMPL #include "erl_afit_alloc.h" +#define GET_ERL_AOFF_ALLOC_IMPL +#include "erl_ao_firstfit_alloc.h" + #define ERTS_ALC_DEFAULT_MAX_THR_PREF 16 @@ -85,15 +88,19 @@ typedef union { char align_bfa[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(BFAllctr_t))]; AFAllctr_t afa; char align_afa[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(AFAllctr_t))]; + AOFFAllctr_t aoffa; + char align_aoffa[ERTS_ALC_CACHE_LINE_ALIGN_SIZE(sizeof(AOFFAllctr_t))]; } ErtsAllocatorState_t; -static ErtsAllocatorState_t sl_alloc_state; +static ErtsAllocatorState_t sbmbc_alloc_state; static ErtsAllocatorState_t std_alloc_state; static ErtsAllocatorState_t ll_alloc_state; #if HALFWORD_HEAP -static ErtsAllocatorState_t std_alloc_low_state; -static ErtsAllocatorState_t ll_alloc_low_state; +static ErtsAllocatorState_t sbmbc_low_alloc_state; +static ErtsAllocatorState_t std_low_alloc_state; +static ErtsAllocatorState_t ll_low_alloc_state; #endif +static ErtsAllocatorState_t sl_alloc_state; static ErtsAllocatorState_t temp_alloc_state; static ErtsAllocatorState_t eheap_alloc_state; static ErtsAllocatorState_t binary_alloc_state; @@ -120,7 +127,8 @@ static void *fix_core_alloc(Uint size) enum allctr_type { GOODFIT, BESTFIT, - AFIT + AFIT, + AOFIRSTFIT }; struct au_init { @@ -132,6 +140,7 @@ struct au_init { GFAllctrInit_t gf; BFAllctrInit_t bf; AFAllctrInit_t af; + AOFFAllctrInit_t aoff; } init; struct { int mmbcs; @@ -145,7 +154,8 @@ struct au_init { ERTS_DEFAULT_ALLCTR_INIT, \ ERTS_DEFAULT_GF_ALLCTR_INIT, \ ERTS_DEFAULT_BF_ALLCTR_INIT, \ - ERTS_DEFAULT_AF_ALLCTR_INIT \ + ERTS_DEFAULT_AF_ALLCTR_INIT, \ + ERTS_DEFAULT_AOFF_ALLCTR_INIT \ } typedef struct { @@ -162,6 +172,7 @@ typedef struct { char *mtrace; char *nodename; } instr; + struct au_init sbmbc_alloc; struct au_init sl_alloc; struct au_init std_alloc; struct au_init ll_alloc; @@ -171,8 +182,9 @@ typedef struct { struct au_init ets_alloc; struct au_init driver_alloc; #if HALFWORD_HEAP - struct au_init std_alloc_low; - struct au_init ll_alloc_low; + struct au_init sbmbc_low_alloc; + struct au_init std_low_alloc; + struct au_init ll_low_alloc; #endif } erts_alc_hndl_args_init_t; @@ -185,6 +197,34 @@ do { \ } while (0) static void +set_default_sbmbc_alloc_opts(struct au_init *ip) +{ + SET_DEFAULT_ALLOC_OPTS(ip); + ip->enable = 0; + ip->thr_spec = 0; + ip->atype = BESTFIT; + ip->init.bf.ao = 1; + ip->init.util.ramv = 0; + ip->init.util.mmsbc = 0; + ip->init.util.mmmbc = 500; + ip->init.util.sbct = ~((UWord) 0); + ip->init.util.name_prefix = "sbmbc_"; + ip->init.util.alloc_no = ERTS_ALC_A_SBMBC; +#ifndef SMALL_MEMORY + ip->init.util.mmbcs = 2*1024*1024; /* Main carrier size */ +#else + ip->init.util.mmbcs = 1*1024*1024; /* Main carrier size */ +#endif + ip->init.util.ts = ERTS_ALC_MTA_SBMBC; + ip->init.util.asbcst = 0; + ip->init.util.rsbcst = 0; + ip->init.util.rsbcmt = 0; + ip->init.util.rmbcmt = 0; + ip->init.util.sbmbct = 0; + ip->init.util.sbmbcs = 0; +} + +static void set_default_sl_alloc_opts(struct au_init *ip) { SET_DEFAULT_ALLOC_OPTS(ip); @@ -202,6 +242,7 @@ set_default_sl_alloc_opts(struct au_init *ip) ip->init.util.ts = ERTS_ALC_MTA_SHORT_LIVED; ip->init.util.rsbcst = 80; #if HALFWORD_HEAP + ip->init.util.force = 1; ip->init.util.low_mem = 1; #endif @@ -249,6 +290,8 @@ set_default_ll_alloc_opts(struct au_init *ip) ip->init.util.rsbcst = 0; ip->init.util.rsbcmt = 0; ip->init.util.rmbcmt = 0; + ip->init.util.sbmbct = 0; + ip->init.util.sbmbcs = 0; } static void @@ -269,6 +312,7 @@ set_default_temp_alloc_opts(struct au_init *ip) ip->init.util.rsbcst = 90; ip->init.util.rmbcmt = 100; #if HALFWORD_HEAP + ip->init.util.force = 1; ip->init.util.low_mem = 1; #endif } @@ -291,6 +335,7 @@ set_default_eheap_alloc_opts(struct au_init *ip) ip->init.util.ts = ERTS_ALC_MTA_EHEAP; ip->init.util.rsbcst = 50; #if HALFWORD_HEAP + ip->init.util.force = 1; ip->init.util.low_mem = 1; #endif } @@ -436,10 +481,13 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) hdbg_init(); #endif + erts_have_sbmbc_alloc = 0; + erts_sys_alloc_init(); init_thr_ix(erts_no_schedulers); erts_init_utils_mem(); + set_default_sbmbc_alloc_opts(&init.sbmbc_alloc); set_default_sl_alloc_opts(&init.sl_alloc); set_default_std_alloc_opts(&init.std_alloc); set_default_ll_alloc_opts(&init.ll_alloc); @@ -453,6 +501,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) handle_args(argc, argv, &init); if (erts_no_schedulers <= 1) { + init.sbmbc_alloc.thr_spec = 0; init.sl_alloc.thr_spec = 0; init.std_alloc.thr_spec = 0; init.ll_alloc.thr_spec = 0; @@ -464,6 +513,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) if (init.erts_alloc_config) { /* Adjust flags that erts_alloc_config won't like */ + init.sbmbc_alloc.thr_spec = 0; init.temp_alloc.thr_spec = 0; init.sl_alloc.thr_spec = 0; init.std_alloc.thr_spec = 0; @@ -480,6 +530,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) init.temp_alloc.thr_spec = erts_no_schedulers; /* Others must use thread preferred interface */ + adjust_tpref(&init.sbmbc_alloc, erts_no_schedulers); adjust_tpref(&init.sl_alloc, erts_no_schedulers); adjust_tpref(&init.std_alloc, erts_no_schedulers); adjust_tpref(&init.ll_alloc, erts_no_schedulers); @@ -497,6 +548,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) * The following allocators cannot be run with afit strategy. * Make sure they don't... */ + refuse_af_strategy(&init.sbmbc_alloc); refuse_af_strategy(&init.sl_alloc); refuse_af_strategy(&init.std_alloc); refuse_af_strategy(&init.ll_alloc); @@ -518,6 +570,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) erts_afalc_init(); erts_bfalc_init(); erts_gfalc_init(); + erts_aoffalc_init(); for (i = ERTS_ALC_A_MIN; i <= ERTS_ALC_A_MAX; i++) { erts_allctrs[i].alloc = NULL; @@ -551,19 +604,30 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) #if HALFWORD_HEAP /* Init low memory variants by cloning */ - init.std_alloc_low = init.std_alloc; - init.std_alloc_low.init.util.alloc_no = ERTS_ALC_A_STANDARD_LOW; - init.std_alloc_low.init.util.low_mem = 1; - - init.ll_alloc_low = init.ll_alloc; - init.ll_alloc_low.init.util.alloc_no = ERTS_ALC_A_LONG_LIVED_LOW; - init.ll_alloc_low.init.util.low_mem = 1; - - set_au_allocator(ERTS_ALC_A_STANDARD_LOW, &init.std_alloc_low); - set_au_allocator(ERTS_ALC_A_LONG_LIVED_LOW, &init.ll_alloc_low); + init.sbmbc_low_alloc = init.sbmbc_alloc; + init.sbmbc_low_alloc.init.util.name_prefix = "sbmbc_low_"; + init.sbmbc_low_alloc.init.util.alloc_no = ERTS_ALC_A_SBMBC_LOW; + init.sbmbc_low_alloc.init.util.low_mem = 1; + + init.std_low_alloc = init.std_alloc; + init.std_low_alloc.init.util.name_prefix = "std_low_"; + init.std_low_alloc.init.util.alloc_no = ERTS_ALC_A_STANDARD_LOW; + init.std_low_alloc.init.util.force = 1; + init.std_low_alloc.init.util.low_mem = 1; + + init.ll_low_alloc = init.ll_alloc; + init.ll_low_alloc.init.util.name_prefix = "ll_low_"; + init.ll_low_alloc.init.util.alloc_no = ERTS_ALC_A_LONG_LIVED_LOW; + init.ll_low_alloc.init.util.force = 1; + init.ll_low_alloc.init.util.low_mem = 1; + + set_au_allocator(ERTS_ALC_A_SBMBC_LOW, &init.sbmbc_low_alloc); + set_au_allocator(ERTS_ALC_A_STANDARD_LOW, &init.std_low_alloc); + set_au_allocator(ERTS_ALC_A_LONG_LIVED_LOW, &init.ll_low_alloc); #endif /* HALFWORD */ set_au_allocator(ERTS_ALC_A_TEMPORARY, &init.temp_alloc); + set_au_allocator(ERTS_ALC_A_SBMBC, &init.sbmbc_alloc); set_au_allocator(ERTS_ALC_A_SHORT_LIVED, &init.sl_alloc); set_au_allocator(ERTS_ALC_A_STANDARD, &init.std_alloc); set_au_allocator(ERTS_ALC_A_LONG_LIVED, &init.ll_alloc); @@ -593,6 +657,20 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) erts_mtrace_init(init.instr.mtrace, init.instr.nodename); + /* sbmbc_alloc() needs to be started first */ + start_au_allocator(ERTS_ALC_A_SBMBC, + &init.sbmbc_alloc, + &sbmbc_alloc_state); +#if HALFWORD_HEAP + start_au_allocator(ERTS_ALC_A_SBMBC_LOW, + &init.sbmbc_low_alloc, + &sbmbc_low_alloc_state); + erts_have_sbmbc_alloc = (init.sbmbc_alloc.enable + && init.sbmbc_low_alloc.enable); +#else + erts_have_sbmbc_alloc = init.sbmbc_alloc.enable; +#endif + start_au_allocator(ERTS_ALC_A_TEMPORARY, &init.temp_alloc, &temp_alloc_state); @@ -610,11 +688,11 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) &ll_alloc_state); #if HALFWORD_HEAP start_au_allocator(ERTS_ALC_A_LONG_LIVED_LOW, - &init.ll_alloc_low, - &ll_alloc_low_state); + &init.ll_low_alloc, + &ll_low_alloc_state); start_au_allocator(ERTS_ALC_A_STANDARD_LOW, - &init.std_alloc_low, - &std_alloc_low_state); + &init.std_low_alloc, + &std_low_alloc_state); #endif start_au_allocator(ERTS_ALC_A_EHEAP, &init.eheap_alloc, @@ -680,14 +758,11 @@ set_au_allocator(ErtsAlcType_t alctr_n, struct au_init *init) ErtsAllocatorInfo_t *ai = &erts_allctrs_info[alctr_n]; ErtsAllocatorThrSpec_t *tspec = &erts_allctr_thr_spec[alctr_n]; -#if HALFWORD_HEAP - /* If halfword heap, silently ignore any disabling of internal - * allocators for low memory + /* + * Some allocators are forced on if halfword heap is used. */ - if (init->init.util.low_mem) { + if (init->init.util.force) init->enable = 1; - } -#endif if (!init->enable) { af->alloc = erts_sys_alloc; @@ -837,6 +912,12 @@ start_au_allocator(ErtsAlcType_t alctr_n, &init->init.af, &init->init.util); break; + case AOFIRSTFIT: + as = (void *) erts_aoffalc_start((AOFFAllctr_t *) as0, + &init->init.aoff, + &init->init.util); + break; + default: as = NULL; ASSERT(0); @@ -947,6 +1028,20 @@ get_kb_value(char *param_end, char** argv, int* ip) } static Uint +get_byte_value(char *param_end, char** argv, int* ip) +{ + Sint tmp; + char *rest; + char *param = argv[*ip]+1; + char *value = get_value(param_end, argv, ip); + errno = 0; + tmp = (Sint) strtol(value, &rest, 10); + if (errno != 0 || rest == value || tmp < 0) + bad_value(param, param_end, value); + return (Uint) tmp; +} + +static Uint get_amount_value(char *param_end, char** argv, int* ip) { Sint tmp; @@ -1017,6 +1112,9 @@ handle_au_arg(struct au_init *auip, else if (strcmp("af", alg) == 0) { auip->atype = AFIT; } + else if (strcmp("aoff", alg) == 0) { + auip->atype = AOFIRSTFIT; + } else { bad_value(param, sub_param + 1, alg); } @@ -1085,6 +1183,12 @@ handle_au_arg(struct au_init *auip, if(has_prefix("sbct", sub_param)) { auip->init.util.sbct = get_kb_value(sub_param + 4, argv, ip); } + else if (has_prefix("sbmbcs", sub_param)) { + auip->init.util.sbmbcs = get_byte_value(sub_param + 6, argv, ip); + } + else if (has_prefix("sbmbct", sub_param)) { + auip->init.util.sbmbct = get_byte_value(sub_param + 6, argv, ip); + } else if (has_prefix("smbcs", sub_param)) { auip->default_.smbcs = 0; auip->init.util.smbcs = get_kb_value(sub_param + 5, argv, ip); @@ -1123,6 +1227,7 @@ static void handle_args(int *argc, char **argv, erts_alc_hndl_args_init_t *init) { struct au_init *aui[] = { + &init->sbmbc_alloc, &init->binary_alloc, &init->std_alloc, &init->ets_alloc, @@ -1150,6 +1255,9 @@ handle_args(int *argc, char **argv, erts_alc_hndl_args_init_t *init) case 'B': handle_au_arg(&init->binary_alloc, &argv[i][3], argv, &i); break; + case 'C': + handle_au_arg(&init->sbmbc_alloc, &argv[i][3], argv, &i); + break; case 'D': handle_au_arg(&init->std_alloc, &argv[i][3], argv, &i); break; @@ -1856,12 +1964,16 @@ erts_memory(int *print_to_p, void *print_to_arg, void *proc, Eterm earg) return am_badarg; } - /* All alloc_util allocators *have* to be enabled */ + /* All alloc_util allocators except sbmbc_alloc *have* to be enabled */ for (ai = ERTS_ALC_A_MIN; ai <= ERTS_ALC_A_MAX; ai++) { switch (ai) { case ERTS_ALC_A_SYSTEM: case ERTS_ALC_A_FIXED_SIZE: + case ERTS_ALC_A_SBMBC: +#if HALFWORD_HEAP + case ERTS_ALC_A_SBMBC_LOW: +#endif break; default: if (!erts_allctrs_info[ai].enabled @@ -1901,6 +2013,12 @@ erts_memory(int *print_to_p, void *print_to_arg, void *proc, Eterm earg) * Often not thread safe and usually never * contain any allocated memory. */ + case ERTS_ALC_A_SBMBC: + /* Included in other allocators */ +#if HALFWORD_HEAP + case ERTS_ALC_A_SBMBC_LOW: + /* Included in other allocators */ +#endif continue; case ERTS_ALC_A_EHEAP: save = &size.processes; @@ -2882,6 +3000,7 @@ unsigned long erts_alc_test(unsigned long op, case 0x2: return erts_bfalc_test(op, a1, a2); case 0x3: return erts_afalc_test(op, a1, a2); case 0x4: return erts_mseg_test(op, a1, a2, a3); + case 0x5: return erts_aoffalc_test(op, a1, a2); case 0xf: switch (op) { case 0xf00: @@ -2925,6 +3044,7 @@ unsigned long erts_alc_test(unsigned long op, init.atype = GOODFIT; init.init.util.name_prefix = (char *) a1; init.init.util.ts = a2 ? 1 : 0; + init.init.util.sbmbct = 0; if ((char **) a3) { char **argv = (char **) a3; @@ -2960,6 +3080,14 @@ unsigned long erts_alc_test(unsigned long op, &init.init.af, &init.init.util); break; + case AOFIRSTFIT: + allctr = erts_aoffalc_start((AOFFAllctr_t *) + erts_alloc(ERTS_ALC_T_UNDEF, + sizeof(AOFFAllctr_t)), + &init.init.aoff, + &init.init.util); + break; + default: ASSERT(0); allctr = NULL; diff --git a/erts/emulator/beam/erl_alloc.h b/erts/emulator/beam/erl_alloc.h index ce792d4d17..c35a60da22 100644 --- a/erts/emulator/beam/erl_alloc.h +++ b/erts/emulator/beam/erl_alloc.h @@ -99,6 +99,14 @@ unsigned long erts_alc_test(unsigned long, #define ERTS_ALC_MIN_LONG_LIVED_TIME (10*60*1000) +#if HALFWORD_HEAP +#define ERTS_IS_SBMBC_ALLOCATOR_NO__(NO) \ + ((NO) == ERTS_ALC_A_SBMBC || (NO) == ERTS_ALC_A_SBMBC_LOW) +#else +#define ERTS_IS_SBMBC_ALLOCATOR_NO__(NO) \ + ((NO) == ERTS_ALC_A_SBMBC) +#endif + typedef struct { int alloc_util; int enabled; diff --git a/erts/emulator/beam/erl_alloc.types b/erts/emulator/beam/erl_alloc.types index c6cc0e1fac..eda0831441 100644 --- a/erts/emulator/beam/erl_alloc.types +++ b/erts/emulator/beam/erl_alloc.types @@ -65,6 +65,11 @@ allocator SYSTEM true sys_alloc +allocator SBMBC true sbmbc_alloc ++if halfword +allocator SBMBC_LOW true sbmbc_low_alloc ++endif + +if smp allocator TEMPORARY true temp_alloc @@ -76,8 +81,8 @@ allocator ETS true ets_alloc allocator FIXED_SIZE true fix_alloc +if halfword -allocator LONG_LIVED_LOW true ll_alloc_low -allocator STANDARD_LOW true std_alloc_low +allocator LONG_LIVED_LOW true ll_low_alloc +allocator STANDARD_LOW true std_low_alloc +endif +else # Non smp build @@ -91,8 +96,8 @@ allocator ETS false ets_alloc allocator FIXED_SIZE false fix_alloc +if halfword -allocator LONG_LIVED_LOW false ll_alloc_low -allocator STANDARD_LOW false std_alloc_low +allocator LONG_LIVED_LOW false ll_low_alloc +allocator STANDARD_LOW false std_low_alloc +endif +endif @@ -134,6 +139,7 @@ class SYSTEM system_data # # <TYPE> <ALLOCATOR> <CLASS> <DESCRIPTION> +type SBMBC SBMBC SYSTEM small_block_mbc type PROC FIXED_SIZE PROCESSES proc type ATOM FIXED_SIZE ATOM atom_entry type MODULE FIXED_SIZE CODE module_entry @@ -330,6 +336,7 @@ type SSB SHORT_LIVED PROCESSES ssb +if halfword +type SBMBC_LOW SBMBC_LOW SYSTEM small_block_mbc_low type DDLL_PROCESS STANDARD_LOW SYSTEM ddll_processes type MONITOR_LH STANDARD_LOW PROCESSES monitor_lh type NLINK_LH STANDARD_LOW PROCESSES nlink_lh diff --git a/erts/emulator/beam/erl_alloc_util.c b/erts/emulator/beam/erl_alloc_util.c index cc04ef65bf..d51ed0c36d 100644 --- a/erts/emulator/beam/erl_alloc_util.c +++ b/erts/emulator/beam/erl_alloc_util.c @@ -66,6 +66,7 @@ static int atoms_initialized = 0; static int initialized = 0; +int erts_have_sbmbc_alloc; #if HAVE_ERTS_MSEG @@ -85,8 +86,6 @@ static int initialized = 0; #undef ASSERT #define ASSERT ASSERT_EXPR -#define ERTS_ALCU_FLG_FAIL_REALLOC_MOVE ((UWord) 1) - #if 0 /* Can be useful for debugging */ #define MBC_REALLOC_ALWAYS_MOVES @@ -275,14 +274,26 @@ static void check_blk_carrier(Allctr_t *, Block_t *); #ifdef DEBUG #define DEBUG_CHECK_CARRIER_NO_SZ(AP) \ - ASSERT(((AP)->sbcs.curr_mseg.no && (AP)->sbcs.curr_mseg.size) \ - || (!(AP)->sbcs.curr_mseg.no && !(AP)->sbcs.curr_mseg.size));\ - ASSERT(((AP)->sbcs.curr_sys_alloc.no && (AP)->sbcs.curr_sys_alloc.size)\ - || (!(AP)->sbcs.curr_sys_alloc.no && !(AP)->sbcs.curr_sys_alloc.size));\ - ASSERT(((AP)->mbcs.curr_mseg.no && (AP)->mbcs.curr_mseg.size) \ - || (!(AP)->mbcs.curr_mseg.no && !(AP)->mbcs.curr_mseg.size));\ - ASSERT(((AP)->mbcs.curr_sys_alloc.no && (AP)->mbcs.curr_sys_alloc.size)\ - || (!(AP)->mbcs.curr_sys_alloc.no && !(AP)->mbcs.curr_sys_alloc.size)) + ASSERT(((AP)->sbcs.curr.norm.mseg.no \ + && (AP)->sbcs.curr.norm.mseg.size) \ + || (!(AP)->sbcs.curr.norm.mseg.no \ + && !(AP)->sbcs.curr.norm.mseg.size)); \ + ASSERT(((AP)->sbcs.curr.norm.sys_alloc.no \ + && (AP)->sbcs.curr.norm.sys_alloc.size) \ + || (!(AP)->sbcs.curr.norm.sys_alloc.no \ + && !(AP)->sbcs.curr.norm.sys_alloc.size)); \ + ASSERT(((AP)->mbcs.curr.norm.mseg.no \ + && (AP)->mbcs.curr.norm.mseg.size) \ + || (!(AP)->mbcs.curr.norm.mseg.no \ + && !(AP)->mbcs.curr.norm.mseg.size)); \ + ASSERT(((AP)->mbcs.curr.norm.sys_alloc.no \ + && (AP)->mbcs.curr.norm.sys_alloc.size) \ + || (!(AP)->mbcs.curr.norm.sys_alloc.no \ + && !(AP)->mbcs.curr.norm.sys_alloc.size)); \ + ASSERT(((AP)->sbmbcs.curr.small_block.no \ + && (AP)->sbmbcs.curr.small_block.size) \ + || (!(AP)->sbmbcs.curr.small_block.no \ + && !(AP)->sbmbcs.curr.small_block.size)) #else #define DEBUG_CHECK_CARRIER_NO_SZ(AP) @@ -292,27 +303,27 @@ static void check_blk_carrier(Allctr_t *, Block_t *); (AP)->sbcs.blocks.curr.size += (BSZ); \ if ((AP)->sbcs.blocks.max.size < (AP)->sbcs.blocks.curr.size) \ (AP)->sbcs.blocks.max.size = (AP)->sbcs.blocks.curr.size; \ - if ((AP)->sbcs.max.no < ((AP)->sbcs.curr_mseg.no \ - + (AP)->sbcs.curr_sys_alloc.no)) \ - (AP)->sbcs.max.no = ((AP)->sbcs.curr_mseg.no \ - + (AP)->sbcs.curr_sys_alloc.no); \ - if ((AP)->sbcs.max.size < ((AP)->sbcs.curr_mseg.size \ - + (AP)->sbcs.curr_sys_alloc.size)) \ - (AP)->sbcs.max.size = ((AP)->sbcs.curr_mseg.size \ - + (AP)->sbcs.curr_sys_alloc.size) + if ((AP)->sbcs.max.no < ((AP)->sbcs.curr.norm.mseg.no \ + + (AP)->sbcs.curr.norm.sys_alloc.no)) \ + (AP)->sbcs.max.no = ((AP)->sbcs.curr.norm.mseg.no \ + + (AP)->sbcs.curr.norm.sys_alloc.no); \ + if ((AP)->sbcs.max.size < ((AP)->sbcs.curr.norm.mseg.size \ + + (AP)->sbcs.curr.norm.sys_alloc.size)) \ + (AP)->sbcs.max.size = ((AP)->sbcs.curr.norm.mseg.size \ + + (AP)->sbcs.curr.norm.sys_alloc.size) #define STAT_MSEG_SBC_ALLOC(AP, CSZ, BSZ) \ do { \ - (AP)->sbcs.curr_mseg.no++; \ - (AP)->sbcs.curr_mseg.size += (CSZ); \ + (AP)->sbcs.curr.norm.mseg.no++; \ + (AP)->sbcs.curr.norm.mseg.size += (CSZ); \ STAT_SBC_ALLOC((AP), (BSZ)); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) #define STAT_SYS_ALLOC_SBC_ALLOC(AP, CSZ, BSZ) \ do { \ - (AP)->sbcs.curr_sys_alloc.no++; \ - (AP)->sbcs.curr_sys_alloc.size += (CSZ); \ + (AP)->sbcs.curr.norm.sys_alloc.no++; \ + (AP)->sbcs.curr.norm.sys_alloc.size += (CSZ); \ STAT_SBC_ALLOC((AP), (BSZ)); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) @@ -324,85 +335,111 @@ do { \ #define STAT_MSEG_SBC_FREE(AP, CSZ, BSZ) \ do { \ - ASSERT((AP)->sbcs.curr_mseg.no > 0); \ - (AP)->sbcs.curr_mseg.no--; \ - ASSERT((AP)->sbcs.curr_mseg.size >= (CSZ)); \ - (AP)->sbcs.curr_mseg.size -= (CSZ); \ + ASSERT((AP)->sbcs.curr.norm.mseg.no > 0); \ + (AP)->sbcs.curr.norm.mseg.no--; \ + ASSERT((AP)->sbcs.curr.norm.mseg.size >= (CSZ)); \ + (AP)->sbcs.curr.norm.mseg.size -= (CSZ); \ STAT_SBC_FREE((AP), (BSZ)); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) #define STAT_SYS_ALLOC_SBC_FREE(AP, CSZ, BSZ) \ do { \ - ASSERT((AP)->sbcs.curr_sys_alloc.no > 0); \ - (AP)->sbcs.curr_sys_alloc.no--; \ - ASSERT((AP)->sbcs.curr_sys_alloc.size >= (CSZ)); \ - (AP)->sbcs.curr_sys_alloc.size -= (CSZ); \ + ASSERT((AP)->sbcs.curr.norm.sys_alloc.no > 0); \ + (AP)->sbcs.curr.norm.sys_alloc.no--; \ + ASSERT((AP)->sbcs.curr.norm.sys_alloc.size >= (CSZ)); \ + (AP)->sbcs.curr.norm.sys_alloc.size -= (CSZ); \ STAT_SBC_FREE((AP), (BSZ)); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) #define STAT_MBC_ALLOC(AP) \ - if ((AP)->mbcs.max.no < ((AP)->mbcs.curr_mseg.no \ - + (AP)->mbcs.curr_sys_alloc.no)) \ - (AP)->mbcs.max.no = ((AP)->mbcs.curr_mseg.no \ - + (AP)->mbcs.curr_sys_alloc.no); \ - if ((AP)->mbcs.max.size < ((AP)->mbcs.curr_mseg.size \ - + (AP)->mbcs.curr_sys_alloc.size)) \ - (AP)->mbcs.max.size = ((AP)->mbcs.curr_mseg.size \ - + (AP)->mbcs.curr_sys_alloc.size) + if ((AP)->mbcs.max.no < ((AP)->mbcs.curr.norm.mseg.no \ + + (AP)->mbcs.curr.norm.sys_alloc.no)) \ + (AP)->mbcs.max.no = ((AP)->mbcs.curr.norm.mseg.no \ + + (AP)->mbcs.curr.norm.sys_alloc.no); \ + if ((AP)->mbcs.max.size < ((AP)->mbcs.curr.norm.mseg.size \ + + (AP)->mbcs.curr.norm.sys_alloc.size)) \ + (AP)->mbcs.max.size = ((AP)->mbcs.curr.norm.mseg.size \ + + (AP)->mbcs.curr.norm.sys_alloc.size) + +#define STAT_SBMBC_ALLOC(AP, CSZ) \ +do { \ + (AP)->sbmbcs.curr.small_block.no++; \ + (AP)->sbmbcs.curr.small_block.size += (CSZ); \ + if ((AP)->sbmbcs.max.no < (AP)->sbmbcs.curr.small_block.no) \ + (AP)->sbmbcs.max.no = (AP)->sbmbcs.curr.small_block.no; \ + if ((AP)->sbmbcs.max.size < (AP)->sbmbcs.curr.small_block.size) \ + (AP)->sbmbcs.max.size = (AP)->sbmbcs.curr.small_block.size; \ + DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ +} while (0) #define STAT_MSEG_MBC_ALLOC(AP, CSZ) \ do { \ - (AP)->mbcs.curr_mseg.no++; \ - (AP)->mbcs.curr_mseg.size += (CSZ); \ + (AP)->mbcs.curr.norm.mseg.no++; \ + (AP)->mbcs.curr.norm.mseg.size += (CSZ); \ STAT_MBC_ALLOC((AP)); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) #define STAT_SYS_ALLOC_MBC_ALLOC(AP, CSZ) \ do { \ - (AP)->mbcs.curr_sys_alloc.no++; \ - (AP)->mbcs.curr_sys_alloc.size += (CSZ); \ + (AP)->mbcs.curr.norm.sys_alloc.no++; \ + (AP)->mbcs.curr.norm.sys_alloc.size += (CSZ); \ STAT_MBC_ALLOC((AP)); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) +#define STAT_SBMBC_FREE(AP, CSZ) \ +do { \ + ASSERT((AP)->sbmbcs.curr.small_block.no > 0); \ + (AP)->sbmbcs.curr.small_block.no--; \ + ASSERT((AP)->sbmbcs.curr.small_block.size >= (CSZ)); \ + (AP)->sbmbcs.curr.small_block.size -= (CSZ); \ + DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ +} while (0) + #define STAT_MSEG_MBC_FREE(AP, CSZ) \ do { \ - ASSERT((AP)->mbcs.curr_mseg.no > 0); \ - (AP)->mbcs.curr_mseg.no--; \ - ASSERT((AP)->mbcs.curr_mseg.size >= (CSZ)); \ - (AP)->mbcs.curr_mseg.size -= (CSZ); \ + ASSERT((AP)->mbcs.curr.norm.mseg.no > 0); \ + (AP)->mbcs.curr.norm.mseg.no--; \ + ASSERT((AP)->mbcs.curr.norm.mseg.size >= (CSZ)); \ + (AP)->mbcs.curr.norm.mseg.size -= (CSZ); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) #define STAT_SYS_ALLOC_MBC_FREE(AP, CSZ) \ do { \ - ASSERT((AP)->mbcs.curr_sys_alloc.no > 0); \ - (AP)->mbcs.curr_sys_alloc.no--; \ - ASSERT((AP)->mbcs.curr_sys_alloc.size >= (CSZ)); \ - (AP)->mbcs.curr_sys_alloc.size -= (CSZ); \ + ASSERT((AP)->mbcs.curr.norm.sys_alloc.no > 0); \ + (AP)->mbcs.curr.norm.sys_alloc.no--; \ + ASSERT((AP)->mbcs.curr.norm.sys_alloc.size >= (CSZ)); \ + (AP)->mbcs.curr.norm.sys_alloc.size -= (CSZ); \ DEBUG_CHECK_CARRIER_NO_SZ((AP)); \ } while (0) -#define STAT_MBC_BLK_ALLOC(AP, BSZ) \ +#define STAT_MBC_BLK_ALLOC(AP, BSZ, FLGS) \ do { \ - (AP)->mbcs.blocks.curr.no++; \ - if ((AP)->mbcs.blocks.max.no < (AP)->mbcs.blocks.curr.no) \ - (AP)->mbcs.blocks.max.no = (AP)->mbcs.blocks.curr.no; \ - (AP)->mbcs.blocks.curr.size += (BSZ); \ - if ((AP)->mbcs.blocks.max.size < (AP)->mbcs.blocks.curr.size) \ - (AP)->mbcs.blocks.max.size = (AP)->mbcs.blocks.curr.size; \ + CarriersStats_t *cstats__ = (((FLGS) & ERTS_ALCU_FLG_SBMBC) \ + ? &(AP)->sbmbcs \ + : &(AP)->mbcs); \ + cstats__->blocks.curr.no++; \ + if (cstats__->blocks.max.no < cstats__->blocks.curr.no) \ + cstats__->blocks.max.no = cstats__->blocks.curr.no; \ + cstats__->blocks.curr.size += (BSZ); \ + if (cstats__->blocks.max.size < cstats__->blocks.curr.size) \ + cstats__->blocks.max.size = cstats__->blocks.curr.size; \ } while (0) -#define STAT_MBC_BLK_FREE(AP, BSZ) \ +#define STAT_MBC_BLK_FREE(AP, BSZ, FLGS) \ do { \ - ASSERT((AP)->mbcs.blocks.curr.no > 0); \ - (AP)->mbcs.blocks.curr.no--; \ - ASSERT((AP)->mbcs.blocks.curr.size >= (BSZ)); \ - (AP)->mbcs.blocks.curr.size -= (BSZ); \ + CarriersStats_t *cstats__ = (((FLGS) & ERTS_ALCU_FLG_SBMBC) \ + ? &(AP)->sbmbcs \ + : &(AP)->mbcs); \ + ASSERT(cstats__->blocks.curr.no > 0); \ + cstats__->blocks.curr.no--; \ + ASSERT(cstats__->blocks.curr.size >= (BSZ)); \ + cstats__->blocks.curr.size -= (BSZ); \ } while (0) /* Debug stuff... */ @@ -410,7 +447,7 @@ do { \ static UWord carrier_alignment; #define DEBUG_SAVE_ALIGNMENT(C) \ do { \ - UWord algnmnt__ = sizeof(Unit_t) - (((UWord) (C)) % sizeof(Unit_t)); \ + UWord algnmnt__ = sizeof(Unit_t) - (((UWord) (C)) % sizeof(Unit_t));\ carrier_alignment = MIN(carrier_alignment, algnmnt__); \ ASSERT(((UWord) (C)) % sizeof(UWord) == 0); \ } while (0) @@ -524,8 +561,8 @@ static Uint get_next_mbc_size(Allctr_t *allctr) { Uint size; - int cs = (allctr->mbcs.curr_mseg.no - + allctr->mbcs.curr_sys_alloc.no + int cs = (allctr->mbcs.curr.norm.mseg.no + + allctr->mbcs.curr.norm.sys_alloc.no - (allctr->main_carrier ? 1 : 0)); ASSERT(cs >= 0); @@ -609,7 +646,8 @@ unlink_carrier(CarrierList_t *cl, Carrier_t *crr) } } - +static Block_t *create_sbmbc(Allctr_t *allctr, Uint umem_sz); +static void destroy_sbmbc(Allctr_t *allctr, Block_t *blk); static Block_t *create_carrier(Allctr_t *, Uint, UWord); static void destroy_carrier(Allctr_t *, Block_t *); @@ -619,39 +657,57 @@ static void destroy_carrier(Allctr_t *, Block_t *); * block in a sbc. */ static ERTS_INLINE void * -mbc_alloc_block(Allctr_t *allctr, Uint size, Uint *blk_szp) +mbc_alloc_block(Allctr_t *allctr, Uint size, Uint *blk_szp, Uint32 *alcu_flgsp) { Block_t *blk; + Uint get_blk_sz; + Uint sbmbct; ASSERT(size); ASSERT(size < allctr->sbc_threshold); - *blk_szp = UMEMSZ2BLKSZ(allctr, size); + *blk_szp = get_blk_sz = UMEMSZ2BLKSZ(allctr, size); + + sbmbct = allctr->sbmbc_threshold; + if (sbmbct) { + if (get_blk_sz < sbmbct) { + *alcu_flgsp |= ERTS_ALCU_FLG_SBMBC; + if (get_blk_sz + allctr->min_block_size > sbmbct) { + /* Since we use block size to determine if blocks are + located in sbmbc or not... */ + get_blk_sz += allctr->min_block_size; + } + } + } - blk = (*allctr->get_free_block)(allctr, *blk_szp, NULL, 0); + blk = (*allctr->get_free_block)(allctr, get_blk_sz, NULL, 0, *alcu_flgsp); -#if HALFWORD_HEAP if (!blk) { - blk = create_carrier(allctr, *blk_szp, CFLG_MBC|CFLG_FORCE_MSEG); - } + if ((*alcu_flgsp) & ERTS_ALCU_FLG_SBMBC) + blk = create_sbmbc(allctr, get_blk_sz); + else { +#if HALFWORD_HEAP + blk = create_carrier(allctr, get_blk_sz, CFLG_MBC|CFLG_FORCE_MSEG); #else - if (!blk) { - blk = create_carrier(allctr, *blk_szp, CFLG_MBC); - if (!blk) { - /* Emergency! We couldn't create the carrier as we wanted. - Try to place it in a sys_alloced sbc. */ - blk = create_carrier(allctr, - size, - CFLG_SBC|CFLG_FORCE_SIZE|CFLG_FORCE_SYS_ALLOC); + blk = create_carrier(allctr, get_blk_sz, CFLG_MBC); + if (!blk) { + /* Emergency! We couldn't create the carrier as we wanted. + Try to place it in a sys_alloced sbc. */ + blk = create_carrier(allctr, + size, + (CFLG_SBC + | CFLG_FORCE_SIZE + | CFLG_FORCE_SYS_ALLOC)); + } +#endif } } -#endif #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG if (IS_MBC_BLK(blk)) { - (*allctr->link_free_block)(allctr, blk); + (*allctr->link_free_block)(allctr, blk, *alcu_flgsp); HARD_CHECK_BLK_CARRIER(allctr, blk); - (*allctr->unlink_free_block)(allctr, blk); + (*allctr->unlink_free_block)(allctr, blk, *alcu_flgsp); } #endif @@ -664,7 +720,8 @@ mbc_alloc_finalize(Allctr_t *allctr, Uint org_blk_sz, UWord flags, Uint want_blk_sz, - int valid_blk_info) + int valid_blk_info, + Uint32 alcu_flgs) { Uint blk_sz; Uint nxt_blk_sz; @@ -700,7 +757,7 @@ mbc_alloc_finalize(Allctr_t *allctr, SET_PREV_BLK_FREE(nxt_nxt_blk); } } - (*allctr->link_free_block)(allctr, nxt_blk); + (*allctr->link_free_block)(allctr, nxt_blk, alcu_flgs); ASSERT(IS_NOT_LAST_BLK(blk)); ASSERT(IS_FREE_BLK(nxt_blk)); @@ -741,7 +798,7 @@ mbc_alloc_finalize(Allctr_t *allctr, : IS_NOT_LAST_BLK(blk)); } - STAT_MBC_BLK_ALLOC(allctr, blk_sz); + STAT_MBC_BLK_ALLOC(allctr, blk_sz, alcu_flgs); ASSERT(IS_ALLOCED_BLK(blk)); ASSERT(blk_sz == BLK_SZ(blk)); @@ -761,7 +818,8 @@ mbc_alloc(Allctr_t *allctr, Uint size) { Block_t *blk; Uint blk_sz; - blk = mbc_alloc_block(allctr, size, &blk_sz); + Uint32 alcu_flgs = 0; + blk = mbc_alloc_block(allctr, size, &blk_sz, &alcu_flgs); if (!blk) return NULL; if (IS_MBC_BLK(blk)) @@ -770,7 +828,8 @@ mbc_alloc(Allctr_t *allctr, Uint size) BLK_SZ(blk), GET_BLK_HDR_FLGS(blk), blk_sz, - 1); + 1, + alcu_flgs); return BLK2UMEM(blk); } @@ -779,6 +838,7 @@ mbc_free(Allctr_t *allctr, void *p) { Uint is_first_blk; Uint is_last_blk; + Uint32 alcu_flgs = 0; Uint blk_sz; Block_t *blk; Block_t *nxt_blk; @@ -788,13 +848,15 @@ mbc_free(Allctr_t *allctr, void *p) blk = UMEM2BLK(p); blk_sz = BLK_SZ(blk); + if (blk_sz < allctr->sbmbc_threshold) + alcu_flgs |= ERTS_ALCU_FLG_SBMBC; ASSERT(IS_MBC_BLK(blk)); ASSERT(blk_sz >= allctr->min_block_size); HARD_CHECK_BLK_CARRIER(allctr, blk); - STAT_MBC_BLK_FREE(allctr, blk_sz); + STAT_MBC_BLK_FREE(allctr, blk_sz, alcu_flgs); is_first_blk = IS_FIRST_BLK(blk); is_last_blk = IS_LAST_BLK(blk); @@ -802,7 +864,7 @@ mbc_free(Allctr_t *allctr, void *p) if (!is_first_blk && IS_PREV_BLK_FREE(blk)) { /* Coalesce with previous block... */ blk = PREV_BLK(blk); - (*allctr->unlink_free_block)(allctr, blk); + (*allctr->unlink_free_block)(allctr, blk, alcu_flgs); blk_sz += BLK_SZ(blk); is_first_blk = IS_FIRST_BLK(blk); @@ -818,7 +880,7 @@ mbc_free(Allctr_t *allctr, void *p) nxt_blk = NXT_BLK(blk); if (IS_FREE_BLK(nxt_blk)) { /* Coalesce with next block... */ - (*allctr->unlink_free_block)(allctr, nxt_blk); + (*allctr->unlink_free_block)(allctr, nxt_blk, alcu_flgs); blk_sz += BLK_SZ(nxt_blk); SET_BLK_SZ(blk, blk_sz); @@ -850,16 +912,20 @@ mbc_free(Allctr_t *allctr, void *p) if (is_first_blk && is_last_blk - && allctr->main_carrier != FBLK2MBC(allctr, blk)) - destroy_carrier(allctr, blk); + && allctr->main_carrier != FBLK2MBC(allctr, blk)) { + if (alcu_flgs & ERTS_ALCU_FLG_SBMBC) + destroy_sbmbc(allctr, blk); + else + destroy_carrier(allctr, blk); + } else { - (*allctr->link_free_block)(allctr, blk); + (*allctr->link_free_block)(allctr, blk, alcu_flgs); HARD_CHECK_BLK_CARRIER(allctr, blk); } } static void * -mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) +mbc_realloc(Allctr_t *allctr, void *p, Uint size, Uint32 alcu_flgs) { void *new_p; Uint old_blk_sz; @@ -867,7 +933,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) #ifndef MBC_REALLOC_ALWAYS_MOVES Block_t *new_blk, *cand_blk; Uint cand_blk_sz; - Uint blk_sz; + Uint blk_sz, get_blk_sz; Block_t *nxt_blk; Uint nxt_blk_sz; Uint is_last_blk; @@ -883,10 +949,16 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) ASSERT(old_blk_sz >= allctr->min_block_size); #ifdef MBC_REALLOC_ALWAYS_MOVES - if (flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) + if (alcu_flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) return NULL; #else /* !MBC_REALLOC_ALWAYS_MOVES */ - blk_sz = UMEMSZ2BLKSZ(allctr, size); + get_blk_sz = blk_sz = UMEMSZ2BLKSZ(allctr, size); + if ((alcu_flgs & ERTS_ALCU_FLG_SBMBC) + && (blk_sz + allctr->min_block_size > allctr->sbmbc_threshold)) { + /* Since we use block size to determine if blocks are + located in sbmbc or not... */ + get_blk_sz = blk_sz + allctr->min_block_size; + } ASSERT(IS_ALLOCED_BLK(blk)); ASSERT(IS_MBC_BLK(blk)); @@ -901,6 +973,9 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) Uint diff_sz_val = old_blk_sz - blk_sz; Uint old_blk_sz_val = old_blk_sz; + if (get_blk_sz >= old_blk_sz) + return p; + if (diff_sz_val >= (~((Uint) 0) / 100)) { /* div both by 128 */ old_blk_sz_val >>= 7; @@ -909,7 +984,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) /* Avoid fragmentation by moving the block if it is shrunk much */ if (100*diff_sz_val > allctr->mbc_move_threshold*old_blk_sz_val) { - if (flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) + if (alcu_flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) return NULL; cand_blk_sz = old_blk_sz; @@ -926,9 +1001,10 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) } new_blk = (*allctr->get_free_block)(allctr, - blk_sz, + get_blk_sz, cand_blk, - cand_blk_sz); + cand_blk_sz, + alcu_flgs); if (new_blk || cand_blk != blk) goto move_into_new_blk; @@ -952,8 +1028,8 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) nxt_blk_sz, SBH_THIS_FREE|SBH_PREV_ALLOCED|SBH_NOT_LAST_BLK); - STAT_MBC_BLK_FREE(allctr, old_blk_sz); - STAT_MBC_BLK_ALLOC(allctr, blk_sz); + STAT_MBC_BLK_FREE(allctr, old_blk_sz, alcu_flgs); + STAT_MBC_BLK_ALLOC(allctr, blk_sz, alcu_flgs); ASSERT(BLK_SZ(blk) >= allctr->min_block_size); @@ -964,7 +1040,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) if (IS_FREE_BLK(nxt_nxt_blk)) { /* Coalesce with next free block... */ nxt_blk_sz += BLK_SZ(nxt_nxt_blk); - (*allctr->unlink_free_block)(allctr, nxt_nxt_blk); + (*allctr->unlink_free_block)(allctr, nxt_nxt_blk, alcu_flgs); SET_BLK_SZ(nxt_blk, nxt_blk_sz); is_last_blk = IS_LAST_BLK(nxt_nxt_blk); @@ -979,7 +1055,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) } } - (*allctr->link_free_block)(allctr, nxt_blk); + (*allctr->link_free_block)(allctr, nxt_blk, alcu_flgs); ASSERT(IS_ALLOCED_BLK(blk)); @@ -1009,12 +1085,12 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) if (!is_last_blk) { nxt_blk = NXT_BLK(blk); nxt_blk_sz = BLK_SZ(nxt_blk); - if (IS_FREE_BLK(nxt_blk) && blk_sz <= old_blk_sz + nxt_blk_sz) { + if (IS_FREE_BLK(nxt_blk) && get_blk_sz <= old_blk_sz + nxt_blk_sz) { /* Grow into next block... */ HARD_CHECK_BLK_CARRIER(allctr, blk); - (*allctr->unlink_free_block)(allctr, nxt_blk); + (*allctr->unlink_free_block)(allctr, nxt_blk, alcu_flgs); nxt_blk_sz -= blk_sz - old_blk_sz; is_last_blk = IS_LAST_BLK(nxt_blk); @@ -1051,13 +1127,13 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) else SET_BLK_SZ_FTR(nxt_blk, nxt_blk_sz); - (*allctr->link_free_block)(allctr, nxt_blk); + (*allctr->link_free_block)(allctr, nxt_blk, alcu_flgs); ASSERT(IS_FREE_BLK(nxt_blk)); } - STAT_MBC_BLK_FREE(allctr, old_blk_sz); - STAT_MBC_BLK_ALLOC(allctr, blk_sz); + STAT_MBC_BLK_FREE(allctr, old_blk_sz, alcu_flgs); + STAT_MBC_BLK_ALLOC(allctr, blk_sz, alcu_flgs); ASSERT(IS_ALLOCED_BLK(blk)); @@ -1088,7 +1164,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) } } - if (flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) + if (alcu_flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) return NULL; /* Need to grow in another block */ @@ -1108,7 +1184,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) } } - if (cand_blk_sz < blk_sz) { + if (cand_blk_sz < get_blk_sz) { /* We wont fit in cand_blk get a new one */ #endif /* !MBC_REALLOC_ALWAYS_MOVES */ @@ -1127,9 +1203,10 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) /* We will at least fit in cand_blk */ new_blk = (*allctr->get_free_block)(allctr, - blk_sz, + get_blk_sz, cand_blk, - cand_blk_sz); + cand_blk_sz, + alcu_flgs); move_into_new_blk: /* * new_blk, and cand_blk have to be correctly set @@ -1142,7 +1219,8 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) BLK_SZ(new_blk), GET_BLK_HDR_FLGS(new_blk), blk_sz, - 1); + 1, + alcu_flgs); new_p = BLK2UMEM(new_blk); sys_memcpy(new_p, p, MIN(size, old_blk_sz - ABLK_HDR_SZ)); mbc_free(allctr, p); @@ -1164,7 +1242,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) HARD_CHECK_BLK_CARRIER(allctr, blk); - (*allctr->unlink_free_block)(allctr, new_blk); /* prev */ + (*allctr->unlink_free_block)(allctr, new_blk, alcu_flgs); /* prev */ if (is_last_blk) new_blk_flgs |= LAST_BLK_HDR_FLG; @@ -1173,7 +1251,7 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) if (IS_FREE_BLK(nxt_blk)) { new_blk_flgs |= GET_LAST_BLK_HDR_FLG(nxt_blk); new_blk_sz += BLK_SZ(nxt_blk); - (*allctr->unlink_free_block)(allctr, nxt_blk); + (*allctr->unlink_free_block)(allctr, nxt_blk, alcu_flgs); } } @@ -1196,9 +1274,10 @@ mbc_realloc(Allctr_t *allctr, void *p, Uint size, UWord flgs) new_blk_sz, new_blk_flgs, blk_sz, - 0); + 0, + alcu_flgs); - STAT_MBC_BLK_FREE(allctr, old_blk_sz); + STAT_MBC_BLK_FREE(allctr, old_blk_sz, alcu_flgs); return new_p; } @@ -1243,6 +1322,100 @@ do { \ #define CHECK_1BLK_CARRIER(A, SBC, MSEGED, C, CSZ, B, BSZ) #endif +static Block_t * +create_sbmbc(Allctr_t *allctr, Uint umem_sz) +{ + Block_t *blk; + Uint blk_sz; + Uint crr_sz = allctr->sbmbc_size; + Carrier_t *crr; + +#if HALFWORD_HEAP + if (allctr->mseg_opt.low_mem) + crr = erts_alloc(ERTS_ALC_T_SBMBC_LOW, crr_sz); + else +#endif + crr = erts_alloc(ERTS_ALC_T_SBMBC, crr_sz); + + INC_CC(allctr->calls.sbmbc_alloc); + SET_CARRIER_HDR(crr, crr_sz, SCH_SYS_ALLOC|SCH_MBC); + + blk = MBC2FBLK(allctr, crr); + +#ifdef ERTS_ALLOC_UTIL_HARD_DEBUG + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) + crr_sz -= sizeof(UWord); +#endif + + blk_sz = UNIT_FLOOR(crr_sz - allctr->mbc_header_size); + + SET_MBC_BLK_FTR(((UWord *) blk)[-1]); + SET_BLK_HDR(blk, blk_sz, SBH_THIS_FREE|SBH_PREV_FREE|SBH_LAST_BLK); + +#ifdef ERTS_ALLOC_UTIL_HARD_DEBUG + *((Carrier_t **) NXT_BLK(blk)) = crr; +#endif + + link_carrier(&allctr->sbmbc_list, crr); + +#ifdef ERTS_ALLOC_UTIL_HARD_DEBUG + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) + crr_sz += sizeof(UWord); +#endif + + STAT_SBMBC_ALLOC(allctr, crr_sz); + CHECK_1BLK_CARRIER(allctr, 0, 0, crr, crr_sz, blk, blk_sz); +#ifdef ERTS_ALLOC_UTIL_HARD_DEBUG + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) + crr_sz -= sizeof(UWord); +#endif + if (allctr->creating_mbc) + (*allctr->creating_mbc)(allctr, crr, ERTS_ALCU_FLG_SBMBC); + + DEBUG_SAVE_ALIGNMENT(crr); + return blk; +} + +static void +destroy_sbmbc(Allctr_t *allctr, Block_t *blk) +{ + Uint crr_sz; + Carrier_t *crr; + + ASSERT(IS_FIRST_BLK(blk)); + + ASSERT(IS_MBC_BLK(blk)); + + crr = FBLK2MBC(allctr, blk); + crr_sz = CARRIER_SZ(crr); + +#ifdef DEBUG + if (!allctr->stopped) { + ASSERT(IS_LAST_BLK(blk)); + +#ifdef ERTS_ALLOC_UTIL_HARD_DEBUG + (*allctr->link_free_block)(allctr, blk, ERTS_ALCU_FLG_SBMBC); + HARD_CHECK_BLK_CARRIER(allctr, blk); + (*allctr->unlink_free_block)(allctr, blk, ERTS_ALCU_FLG_SBMBC); +#endif + } +#endif + + STAT_SBMBC_FREE(allctr, crr_sz); + + unlink_carrier(&allctr->sbmbc_list, crr); + if (allctr->destroying_mbc) + (*allctr->destroying_mbc)(allctr, crr, ERTS_ALCU_FLG_SBMBC); + + INC_CC(allctr->calls.sbmbc_free); + +#if HALFWORD_HEAP + if (allctr->mseg_opt.low_mem) + erts_free(ERTS_ALC_T_SBMBC_LOW, crr); + else +#endif + erts_free(ERTS_ALC_T_SBMBC, crr); +} static Block_t * create_carrier(Allctr_t *allctr, Uint umem_sz, UWord flags) @@ -1271,11 +1444,11 @@ create_carrier(Allctr_t *allctr, Uint umem_sz, UWord flags) if (erts_mseg_no() >= max_mseg_carriers) goto try_sys_alloc; if (flags & CFLG_SBC) { - if (allctr->sbcs.curr_mseg.no >= allctr->max_mseg_sbcs) + if (allctr->sbcs.curr.norm.mseg.no >= allctr->max_mseg_sbcs) goto try_sys_alloc; } else { - if (allctr->mbcs.curr_mseg.no >= allctr->max_mseg_mbcs) + if (allctr->mbcs.curr.norm.mseg.no >= allctr->max_mseg_mbcs) goto try_sys_alloc; } @@ -1289,7 +1462,7 @@ create_carrier(Allctr_t *allctr, Uint umem_sz, UWord flags) if (crr_sz < allctr->mbc_header_size + blk_sz) crr_sz = allctr->mbc_header_size + blk_sz; #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG - if (sizeof(Unit_t) == sizeof(UWord)) + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) crr_sz += sizeof(UWord); #endif } @@ -1330,7 +1503,7 @@ create_carrier(Allctr_t *allctr, Uint umem_sz, UWord flags) && bcrr_sz < allctr->smallest_mbc_size) bcrr_sz = allctr->smallest_mbc_size; #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG - if (sizeof(Unit_t) == sizeof(UWord)) + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) bcrr_sz += sizeof(UWord); #endif @@ -1385,7 +1558,7 @@ create_carrier(Allctr_t *allctr, Uint umem_sz, UWord flags) blk = MBC2FBLK(allctr, crr); #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG - if (sizeof(Unit_t) == sizeof(UWord)) + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) crr_sz -= sizeof(UWord); #endif @@ -1406,16 +1579,16 @@ create_carrier(Allctr_t *allctr, Uint umem_sz, UWord flags) link_carrier(&allctr->mbc_list, crr); #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG - if (sizeof(Unit_t) == sizeof(UWord)) + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) crr_sz += sizeof(UWord); #endif CHECK_1BLK_CARRIER(allctr, 0, is_mseg, crr, crr_sz, blk, blk_sz); #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG - if (sizeof(Unit_t) == sizeof(UWord)) + if (allctr->mbc_header_size % sizeof(Unit_t) == 0) crr_sz -= sizeof(UWord); #endif if (allctr->creating_mbc) - (*allctr->creating_mbc)(allctr, crr); + (*allctr->creating_mbc)(allctr, crr, 0); } @@ -1595,9 +1768,9 @@ destroy_carrier(Allctr_t *allctr, Block_t *blk) ASSERT(IS_LAST_BLK(blk)); #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG - (*allctr->link_free_block)(allctr, blk); + (*allctr->link_free_block)(allctr, blk, 0); HARD_CHECK_BLK_CARRIER(allctr, blk); - (*allctr->unlink_free_block)(allctr, blk); + (*allctr->unlink_free_block)(allctr, blk, 0); #endif } #endif @@ -1614,7 +1787,7 @@ destroy_carrier(Allctr_t *allctr, Block_t *blk) unlink_carrier(&allctr->mbc_list, crr); if (allctr->destroying_mbc) - (*allctr->destroying_mbc)(allctr, crr); + (*allctr->destroying_mbc)(allctr, crr, 0); } @@ -1658,12 +1831,15 @@ static struct { Eterm lmbcs; Eterm smbcs; Eterm mbcgs; + Eterm sbmbcs; + Eterm sbmbct; #if HAVE_ERTS_MSEG Eterm mmc; #endif Eterm ycs; + /* Eterm sbmbcs; */ Eterm mbcs; Eterm sbcs; Eterm sys_alloc_carriers_size; @@ -1688,6 +1864,8 @@ static struct { Eterm mseg_dealloc; Eterm mseg_realloc; #endif + Eterm sbmbc_alloc; + Eterm sbmbc_free; #ifdef DEBUG Eterm end_of_atoms; #endif @@ -1746,12 +1924,15 @@ init_atoms(Allctr_t *allctr) AM_INIT(lmbcs); AM_INIT(smbcs); AM_INIT(mbcgs); + AM_INIT(sbmbcs); + AM_INIT(sbmbct); #if HAVE_ERTS_MSEG AM_INIT(mmc); #endif AM_INIT(ycs); + /*AM_INIT(sbmbcs);*/ AM_INIT(mbcs); AM_INIT(sbcs); AM_INIT(sys_alloc_carriers_size); @@ -1776,6 +1957,8 @@ init_atoms(Allctr_t *allctr) AM_INIT(mseg_dealloc); AM_INIT(mseg_realloc); #endif + AM_INIT(sbmbc_free); + AM_INIT(sbmbc_alloc); #ifdef DEBUG for (atom = (Eterm *) &am; atom < &am.end_of_atoms; atom++) { @@ -1869,7 +2052,9 @@ sz_info_carriers(Allctr_t *allctr, Uint *szp) { Eterm res = THE_NON_VALUE; - Uint curr_size = cs->curr_mseg.size + cs->curr_sys_alloc.size; + Uint curr_size = (cs == &allctr->sbmbcs + ? cs->curr.small_block.size + : cs->curr.norm.mseg.size + cs->curr.norm.sys_alloc.size); if (print_to_p) { int to = *print_to_p; @@ -1917,8 +2102,17 @@ info_carriers(Allctr_t *allctr, Uint *szp) { Eterm res = THE_NON_VALUE; - Uint curr_no = cs->curr_mseg.no + cs->curr_sys_alloc.no; - Uint curr_size = cs->curr_mseg.size + cs->curr_sys_alloc.size; + Uint curr_no, curr_size; + int small_block = cs == &allctr->sbmbcs; + + if (small_block) { + curr_no = cs->curr.small_block.no; + curr_size = cs->curr.small_block.size; + } + else { + curr_no = cs->curr.norm.mseg.no + cs->curr.norm.sys_alloc.no; + curr_size = cs->curr.norm.mseg.size + cs->curr.norm.sys_alloc.size; + } if (print_to_p) { int to = *print_to_p; @@ -1944,18 +2138,20 @@ info_carriers(Allctr_t *allctr, curr_no, cs->max.no, cs->max_ever.no); + if (!small_block) { #if HAVE_ERTS_MSEG - erts_print(to, - arg, - "%smseg carriers: %bpu\n", - prefix, - cs->curr_mseg.no); -#endif - erts_print(to, - arg, - "%ssys_alloc carriers: %bpu\n", - prefix, - cs->curr_sys_alloc.no); + erts_print(to, + arg, + "%smseg carriers: %bpu\n", + prefix, + cs->curr.norm.mseg.no); +#endif + erts_print(to, + arg, + "%ssys_alloc carriers: %bpu\n", + prefix, + cs->curr.norm.sys_alloc.no); + } erts_print(to, arg, "%scarriers size: %beu %bpu %bpu\n", @@ -1963,43 +2159,49 @@ info_carriers(Allctr_t *allctr, curr_size, cs->max.size, cs->max_ever.size); + if (!small_block) { #if HAVE_ERTS_MSEG - erts_print(to, - arg, - "%smseg carriers size: %bpu\n", - prefix, - cs->curr_mseg.size); -#endif - erts_print(to, - arg, - "%ssys_alloc carriers size: %bpu\n", - prefix, - cs->curr_sys_alloc.size); + erts_print(to, + arg, + "%smseg carriers size: %bpu\n", + prefix, + cs->curr.norm.mseg.size); +#endif + erts_print(to, + arg, + "%ssys_alloc carriers size: %bpu\n", + prefix, + cs->curr.norm.sys_alloc.size); + } } if (hpp || szp) { res = NIL; - add_2tup(hpp, szp, &res, - am.sys_alloc_carriers_size, - bld_unstable_uint(hpp, szp, cs->curr_sys_alloc.size)); + if (!small_block) { + add_2tup(hpp, szp, &res, + am.sys_alloc_carriers_size, + bld_unstable_uint(hpp, szp, cs->curr.norm.sys_alloc.size)); #if HAVE_ERTS_MSEG - add_2tup(hpp, szp, &res, - am.mseg_alloc_carriers_size, - bld_unstable_uint(hpp, szp, cs->curr_mseg.size)); + add_2tup(hpp, szp, &res, + am.mseg_alloc_carriers_size, + bld_unstable_uint(hpp, szp, cs->curr.norm.mseg.size)); #endif + } add_4tup(hpp, szp, &res, am.carriers_size, bld_unstable_uint(hpp, szp, curr_size), bld_unstable_uint(hpp, szp, cs->max.size), bld_unstable_uint(hpp, szp, cs->max_ever.size)); - add_2tup(hpp, szp, &res, - am.sys_alloc_carriers, - bld_unstable_uint(hpp, szp, cs->curr_sys_alloc.no)); + if (!small_block) { + add_2tup(hpp, szp, &res, + am.sys_alloc_carriers, + bld_unstable_uint(hpp, szp, cs->curr.norm.sys_alloc.no)); #if HAVE_ERTS_MSEG - add_2tup(hpp, szp, &res, - am.mseg_alloc_carriers, - bld_unstable_uint(hpp, szp, cs->curr_mseg.no)); + add_2tup(hpp, szp, &res, + am.mseg_alloc_carriers, + bld_unstable_uint(hpp, szp, cs->curr.norm.mseg.no)); #endif + } add_4tup(hpp, szp, &res, am.carriers, bld_unstable_uint(hpp, szp, curr_no), @@ -2077,6 +2279,9 @@ info_calls(Allctr_t *allctr, PRINT_CC_5(to, arg, prefix, "free", allctr->calls.this_free); PRINT_CC_5(to, arg, prefix, "realloc", allctr->calls.this_realloc); + PRINT_CC_4(to, arg, "sbmbc_alloc", allctr->calls.sbmbc_alloc); + PRINT_CC_4(to, arg, "sbmbc_free", allctr->calls.sbmbc_free); + #if HAVE_ERTS_MSEG PRINT_CC_4(to, arg, "mseg_alloc", allctr->calls.mseg_alloc); PRINT_CC_4(to, arg, "mseg_dealloc", allctr->calls.mseg_dealloc); @@ -2128,6 +2333,14 @@ info_calls(Allctr_t *allctr, bld_unstable_uint(hpp, szp, allctr->calls.mseg_alloc.no)); #endif add_3tup(hpp, szp, &res, + am.sbmbc_free, + bld_unstable_uint(hpp, szp, allctr->calls.sbmbc_free.giga_no), + bld_unstable_uint(hpp, szp, allctr->calls.sbmbc_free.no)); + add_3tup(hpp, szp, &res, + am.sbmbc_alloc, + bld_unstable_uint(hpp, szp, allctr->calls.sbmbc_alloc.giga_no), + bld_unstable_uint(hpp, szp, allctr->calls.sbmbc_alloc.no)); + add_3tup(hpp, szp, &res, allctr->name.realloc, bld_unstable_uint(hpp, szp, allctr->calls.this_realloc.giga_no), bld_unstable_uint(hpp, szp, allctr->calls.this_realloc.no)); @@ -2191,7 +2404,9 @@ info_options(Allctr_t *allctr, #endif "option lmbcs: %beu\n" "option smbcs: %beu\n" - "option mbcgs: %beu\n", + "option mbcgs: %beu\n" + "option sbmbcs: %beu\n" + "option sbmbct: %beu\n", topt, allctr->ramv ? "true" : "false", #if HALFWORD_HEAP @@ -2211,7 +2426,9 @@ info_options(Allctr_t *allctr, #endif allctr->largest_mbc_size, allctr->smallest_mbc_size, - allctr->mbc_growth_stages); + allctr->mbc_growth_stages, + allctr->sbmbc_size, + allctr->sbmbc_threshold); } res = (*allctr->info_options)(allctr, "option ", print_to_p, print_to_arg, @@ -2219,6 +2436,12 @@ info_options(Allctr_t *allctr, if (hpp || szp) { add_2tup(hpp, szp, &res, + am.sbmbct, + bld_uint(hpp, szp, allctr->sbmbc_threshold)); + add_2tup(hpp, szp, &res, + am.sbmbcs, + bld_uint(hpp, szp, allctr->sbmbc_size)); + add_2tup(hpp, szp, &res, am.mbcgs, bld_uint(hpp, szp, allctr->mbc_growth_stages)); add_2tup(hpp, szp, &res, @@ -2285,10 +2508,10 @@ update_max_ever_values(CarriersStats_t *cs) static ERTS_INLINE void reset_max_values(CarriersStats_t *cs) { - cs->max.no = cs->curr_mseg.no + cs->curr_sys_alloc.no; - cs->max.size = cs->curr_mseg.size + cs->curr_sys_alloc.size; - cs->blocks.max.no = cs->blocks.curr.no; - cs->blocks.max.size = cs->blocks.curr.size; + cs->max.no = cs->curr.norm.mseg.no + cs->curr.norm.sys_alloc.no; + cs->max.size = cs->curr.norm.mseg.size + cs->curr.norm.sys_alloc.size; + cs->blocks.max.no = cs->blocks.curr.no; + cs->blocks.max.size = cs->blocks.curr.size; } @@ -2367,7 +2590,7 @@ erts_alcu_sz_info(Allctr_t *allctr, Uint **hpp, Uint *szp) { - Eterm res, mbcs, sbcs; + Eterm res, sbmbcs, mbcs, sbcs; res = THE_NON_VALUE; @@ -2389,24 +2612,29 @@ erts_alcu_sz_info(Allctr_t *allctr, /* Update sbc values not continously updated */ allctr->sbcs.blocks.curr.no - = allctr->sbcs.curr_mseg.no + allctr->sbcs.curr_sys_alloc.no; + = allctr->sbcs.curr.norm.mseg.no + allctr->sbcs.curr.norm.sys_alloc.no; allctr->sbcs.blocks.max.no = allctr->sbcs.max.no; + update_max_ever_values(&allctr->sbmbcs); update_max_ever_values(&allctr->mbcs); update_max_ever_values(&allctr->sbcs); - mbcs = sz_info_carriers(allctr, &allctr->mbcs, "mbcs ", print_to_p, - print_to_arg, hpp, szp); - sbcs = sz_info_carriers(allctr, &allctr->sbcs, "sbcs ", print_to_p, - print_to_arg, hpp, szp); + sbmbcs = sz_info_carriers(allctr, &allctr->sbmbcs, "sbmbcs ", print_to_p, + print_to_arg, hpp, szp); + mbcs = sz_info_carriers(allctr, &allctr->mbcs, "mbcs ", print_to_p, + print_to_arg, hpp, szp); + sbcs = sz_info_carriers(allctr, &allctr->sbcs, "sbcs ", print_to_p, + print_to_arg, hpp, szp); if (hpp || szp) { res = NIL; add_2tup(hpp, szp, &res, am.sbcs, sbcs); add_2tup(hpp, szp, &res, am.mbcs, mbcs); + add_2tup(hpp, szp, &res, am.sbmbcs, sbmbcs); } if (begin_max_period) { + reset_max_values(&allctr->sbmbcs); reset_max_values(&allctr->mbcs); reset_max_values(&allctr->sbcs); } @@ -2428,7 +2656,7 @@ erts_alcu_info(Allctr_t *allctr, Uint **hpp, Uint *szp) { - Eterm res, sett, mbcs, sbcs, calls; + Eterm res, sett, sbmbcs, mbcs, sbcs, calls; res = THE_NON_VALUE; @@ -2450,9 +2678,10 @@ erts_alcu_info(Allctr_t *allctr, /* Update sbc values not continously updated */ allctr->sbcs.blocks.curr.no - = allctr->sbcs.curr_mseg.no + allctr->sbcs.curr_sys_alloc.no; + = allctr->sbcs.curr.norm.mseg.no + allctr->sbcs.curr.norm.sys_alloc.no; allctr->sbcs.blocks.max.no = allctr->sbcs.max.no; + update_max_ever_values(&allctr->sbmbcs); update_max_ever_values(&allctr->mbcs); update_max_ever_values(&allctr->sbcs); @@ -2464,11 +2693,13 @@ erts_alcu_info(Allctr_t *allctr, ERTS_ALCU_VSN_STR); } - sett = info_options(allctr, print_to_p, print_to_arg, hpp, szp); - mbcs = info_carriers(allctr, &allctr->mbcs, "mbcs ", print_to_p, - print_to_arg, hpp, szp); - sbcs = info_carriers(allctr, &allctr->sbcs, "sbcs ", print_to_p, - print_to_arg, hpp, szp); + sett = info_options(allctr, print_to_p, print_to_arg, hpp, szp); + sbmbcs = info_carriers(allctr, &allctr->sbmbcs, "sbmbcs ", print_to_p, + print_to_arg, hpp, szp); + mbcs = info_carriers(allctr, &allctr->mbcs, "mbcs ", print_to_p, + print_to_arg, hpp, szp); + sbcs = info_carriers(allctr, &allctr->sbcs, "sbcs ", print_to_p, + print_to_arg, hpp, szp); calls = info_calls(allctr, print_to_p, print_to_arg, hpp, szp); if (hpp || szp) { @@ -2477,6 +2708,7 @@ erts_alcu_info(Allctr_t *allctr, add_2tup(hpp, szp, &res, am.calls, calls); add_2tup(hpp, szp, &res, am.sbcs, sbcs); add_2tup(hpp, szp, &res, am.mbcs, mbcs); + add_2tup(hpp, szp, &res, am.sbmbcs, sbmbcs); add_2tup(hpp, szp, &res, am.options, sett); add_3tup(hpp, szp, &res, am.versions, @@ -2485,6 +2717,7 @@ erts_alcu_info(Allctr_t *allctr, } if (begin_max_period) { + reset_max_values(&allctr->sbmbcs); reset_max_values(&allctr->mbcs); reset_max_values(&allctr->sbcs); } @@ -2508,12 +2741,14 @@ erts_alcu_current_size(Allctr_t *allctr, AllctrSize_t *size) erts_mtx_lock(&allctr->mutex); #endif - size->carriers = allctr->mbcs.curr_mseg.size; - size->carriers += allctr->mbcs.curr_sys_alloc.size; - size->carriers += allctr->sbcs.curr_mseg.size; - size->carriers += allctr->sbcs.curr_sys_alloc.size; + size->carriers = allctr->mbcs.curr.norm.mseg.size; + size->carriers += allctr->mbcs.curr.norm.sys_alloc.size; + size->carriers += allctr->sbmbcs.curr.small_block.size; + size->carriers += allctr->sbcs.curr.norm.mseg.size; + size->carriers += allctr->sbcs.curr.norm.sys_alloc.size; size->blocks = allctr->mbcs.blocks.curr.size; + size->blocks += allctr->sbmbcs.blocks.curr.size; size->blocks += allctr->sbcs.blocks.curr.size; #ifdef USE_THREADS @@ -2725,7 +2960,7 @@ do_erts_alcu_realloc(ErtsAlcType_t type, void *extra, void *p, Uint size, - UWord flgs) + Uint32 alcu_flgs) { Allctr_t *allctr = (Allctr_t *) extra; Block_t *blk; @@ -2758,9 +2993,32 @@ do_erts_alcu_realloc(ErtsAlcType_t type, blk = UMEM2BLK(p); + if (allctr->sbmbc_threshold > 0) { + Uint old_sz, new_sz, lim; + lim = allctr->sbmbc_threshold; + old_sz = BLK_SZ(blk); + new_sz = UMEMSZ2BLKSZ(allctr, size); + if ((old_sz < lim && lim <= new_sz) + || (new_sz < lim && lim <= old_sz)) { + /* *Need* to move it... */ + + INC_CC(allctr->calls.this_realloc); + res = do_erts_alcu_alloc(type, extra, size); + DEC_CC(allctr->calls.this_alloc); + + sys_memcpy(res, p, MIN(size, old_sz - ABLK_HDR_SZ)); + + do_erts_alcu_free(type, extra, p); + DEC_CC(allctr->calls.this_free); + return res; + } + if (old_sz < lim) + alcu_flgs |= ERTS_ALCU_FLG_SBMBC; + } + if (size < allctr->sbc_threshold) { if (IS_MBC_BLK(blk)) - res = mbc_realloc(allctr, p, size, flgs); + res = mbc_realloc(allctr, p, size, alcu_flgs); else { Uint used_sz = allctr->sbc_header_size + ABLK_HDR_SZ + size; Uint crr_sz; @@ -2791,7 +3049,7 @@ do_erts_alcu_realloc(ErtsAlcType_t type, if (100*diff_sz_val < allctr->sbc_move_threshold*crr_sz_val) /* Data won't be copied into a new carrier... */ goto do_carrier_resize; - else if (flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) + else if (alcu_flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) return NULL; res = mbc_alloc(allctr, size); @@ -2814,7 +3072,7 @@ do_erts_alcu_realloc(ErtsAlcType_t type, #endif res = new_blk ? BLK2UMEM(new_blk) : NULL; } - else if (flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) + else if (alcu_flgs & ERTS_ALCU_FLG_FAIL_REALLOC_MOVE) return NULL; else { #if HALFWORD_HEAP @@ -3174,6 +3432,26 @@ erts_alcu_start(Allctr_t *allctr, AllctrInit_t *init) allctr->min_block_size = UNIT_CEILING(allctr->min_block_size + sizeof(UWord)); + + allctr->sbmbc_threshold = init->sbmbct; + + if (!erts_have_sbmbc_alloc + || ERTS_IS_SBMBC_ALLOCATOR_NO__(allctr->alloc_no)) + allctr->sbmbc_threshold = 0; + + if (!allctr->sbmbc_threshold) + allctr->sbmbc_size = 0; + else { + Uint min_size; + allctr->sbmbc_size = init->sbmbcs; + min_size = allctr->sbmbc_threshold; + min_size += allctr->min_block_size; + min_size += allctr->mbc_header_size; + if (allctr->sbmbc_size < min_size) + allctr->sbmbc_size = min_size; + } + + #if HAVE_ERTS_MSEG if (allctr->mseg_opt.abs_shrink_th > ~((UWord) 0) / 100) allctr->mseg_opt.abs_shrink_th = ~((UWord) 0) / 100; @@ -3185,12 +3463,16 @@ erts_alcu_start(Allctr_t *allctr, AllctrInit_t *init) #ifdef ERTS_ENABLE_LOCK_COUNT erts_mtx_init_x_opt(&allctr->mutex, - "alcu_allocator", - make_small(allctr->alloc_no), - ERTS_LCNT_LT_ALLOC); + ERTS_IS_SBMBC_ALLOCATOR_NO__(allctr->alloc_no) + ? "sbmbc_alloc" + : "alcu_allocator", + make_small(allctr->alloc_no), + ERTS_LCNT_LT_ALLOC); #else erts_mtx_init_x(&allctr->mutex, - "alcu_allocator", + ERTS_IS_SBMBC_ALLOCATOR_NO__(allctr->alloc_no) + ? "sbmbc_alloc" + : "alcu_allocator", make_small(allctr->alloc_no)); #endif /*ERTS_ENABLE_LOCK_COUNT*/ @@ -3260,7 +3542,7 @@ erts_alcu_start(Allctr_t *allctr, AllctrInit_t *init) if (!blk) goto error; - (*allctr->link_free_block)(allctr, blk); + (*allctr->link_free_block)(allctr, blk, 0); HARD_CHECK_BLK_CARRIER(allctr, blk); @@ -3290,6 +3572,8 @@ erts_alcu_stop(Allctr_t *allctr) destroy_carrier(allctr, SBC2BLK(allctr, allctr->sbc_list.first)); while (allctr->mbc_list.first) destroy_carrier(allctr, MBC2FBLK(allctr, allctr->mbc_list.first)); + while (allctr->sbmbc_list.first) + destroy_sbmbc(allctr, MBC2FBLK(allctr, allctr->sbmbc_list.first)); #ifdef USE_THREADS if (allctr->thread_safe) @@ -3387,13 +3671,15 @@ erts_alcu_verify_unused(Allctr_t *allctr) { UWord no; - no = allctr->sbcs.curr_mseg.no; - no += allctr->sbcs.curr_sys_alloc.no; + no = allctr->sbcs.curr.norm.mseg.no; + no += allctr->sbcs.curr.norm.sys_alloc.no; no += allctr->mbcs.blocks.curr.no; + no += allctr->sbmbcs.blocks.curr.no; if (no) { UWord sz = allctr->sbcs.blocks.curr.size; sz += allctr->mbcs.blocks.curr.size; + sz += allctr->sbmbcs.blocks.curr.size; erl_exit(ERTS_ABORT_EXIT, "%salloc() used when expected to be unused!\n" "Total amount of blocks allocated: %bpu\n" @@ -3492,7 +3778,7 @@ check_blk_carrier(Allctr_t *allctr, Block_t *iblk) (*allctr->check_block)(allctr, blk, (int) is_free_blk); if (IS_LAST_BLK(blk)) { - carrier_end = ((char *) NXT_BLK(blk)) + sizeof(UWord); + carrier_end = ((char *) NXT_BLK(blk)); mbc = *((Carrier_t **) NXT_BLK(blk)); prev_blk = NULL; blk = MBC2FBLK(allctr, mbc); @@ -3507,9 +3793,9 @@ check_blk_carrier(Allctr_t *allctr, Block_t *iblk) ASSERT(IS_MB_CARRIER(mbc)); ASSERT((((char *) mbc) + allctr->mbc_header_size - + tot_blk_sz - + sizeof(UWord)) == carrier_end); - ASSERT(((char *) mbc) + CARRIER_SZ(mbc) == carrier_end); + + tot_blk_sz) == carrier_end); + ASSERT(((char *) mbc) + CARRIER_SZ(mbc) - sizeof(Unit_t) <= carrier_end + && carrier_end <= ((char *) mbc) + CARRIER_SZ(mbc)); if (allctr->check_mbc) (*allctr->check_mbc)(allctr, mbc); @@ -3523,6 +3809,7 @@ check_blk_carrier(Allctr_t *allctr, Block_t *iblk) cl = &allctr->mbc_list; } +#if 0 /* FIXIT sbmbc */ if (cl->first == crr) { ASSERT(!crr->prev); } @@ -3537,6 +3824,7 @@ check_blk_carrier(Allctr_t *allctr, Block_t *iblk) ASSERT(crr->next); ASSERT(crr->next->prev == crr); } +#endif } #endif diff --git a/erts/emulator/beam/erl_alloc_util.h b/erts/emulator/beam/erl_alloc_util.h index ddf84c086c..fed4d3dbe6 100644 --- a/erts/emulator/beam/erl_alloc_util.h +++ b/erts/emulator/beam/erl_alloc_util.h @@ -34,6 +34,7 @@ typedef struct { typedef struct { char *name_prefix; ErtsAlcType_t alloc_no; + int force; int ts; int tspec; int tpref; @@ -50,6 +51,8 @@ typedef struct { UWord lmbcs; UWord smbcs; UWord mbcgs; + UWord sbmbct; + UWord sbmbcs; } AllctrInit_t; typedef struct { @@ -67,6 +70,7 @@ typedef struct { #define ERTS_DEFAULT_ALLCTR_INIT { \ NULL, \ ERTS_ALC_A_INVALID, /* (number) alloc_no: allocator number */\ + 0, /* (bool) force: force enabled */\ 1, /* (bool) ts: thread safe */\ 0, /* (bool) tspec: thread specific */\ 0, /* (bool) tpref: thread preferred */\ @@ -82,7 +86,9 @@ typedef struct { 10, /* (amount) mmmbc: max mseg mbcs */\ 10*1024*1024, /* (bytes) lmbcs: largest mbc size */\ 1024*1024, /* (bytes) smbcs: smallest mbc size */\ - 10 /* (amount) mbcgs: mbc growth stages */\ + 10, /* (amount) mbcgs: mbc growth stages */\ + 256, /* (bytes) sbmbct: small block mbc threshold */\ + 8*1024 /* (bytes) sbmbcs: small block mbc size */\ } #else /* if SMALL_MEMORY */ @@ -95,6 +101,7 @@ typedef struct { #define ERTS_DEFAULT_ALLCTR_INIT { \ NULL, \ ERTS_ALC_A_INVALID, /* (number) alloc_no: allocator number */\ + 0, /* (bool) force: force enabled */\ 1, /* (bool) ts: thread safe */\ 0, /* (bool) tspec: thread specific */\ 0, /* (bool) tpref: thread preferred */\ @@ -109,7 +116,9 @@ typedef struct { 10, /* (amount) mmmbc: max mseg mbcs */\ 1024*1024, /* (bytes) lmbcs: largest mbc size */\ 128*1024, /* (bytes) smbcs: smallest mbc size */\ - 10 /* (amount) mbcgs: mbc growth stages */\ + 10, /* (amount) mbcgs: mbc growth stages */\ + 256, /* (bytes) sbmbct: small block mbc threshold */\ + 8*1024 /* (bytes) sbmbcs: small block mbc size */\ } #endif @@ -144,6 +153,9 @@ void erts_alcu_current_size(Allctr_t *, AllctrSize_t *); #if defined(GET_ERL_ALLOC_UTIL_IMPL) && !defined(ERL_ALLOC_UTIL_IMPL__) #define ERL_ALLOC_UTIL_IMPL__ +#define ERTS_ALCU_FLG_FAIL_REALLOC_MOVE (((Uint32) 1) << 0) +#define ERTS_ALCU_FLG_SBMBC (((Uint32) 1) << 1) + #ifdef USE_THREADS #define ERL_THREADS_EMU_INTERNAL__ #include "erl_threads.h" @@ -188,6 +200,8 @@ void erts_alcu_current_size(Allctr_t *, AllctrSize_t *); #define CARRIER_SZ(C) \ ((C)->chdr & SZ_MASK) +extern int erts_have_sbmbc_alloc; + typedef union {char c[8]; long l; double d;} Unit_t; typedef struct Carrier_t_ Carrier_t; @@ -216,8 +230,13 @@ typedef struct { } StatValues_t; typedef struct { - StatValues_t curr_mseg; - StatValues_t curr_sys_alloc; + union { + struct { + StatValues_t mseg; + StatValues_t sys_alloc; + } norm; + StatValues_t small_block; + } curr; StatValues_t max; StatValues_t max_ever; struct { @@ -257,6 +276,8 @@ struct Allctr_t_ { Uint largest_mbc_size; Uint smallest_mbc_size; Uint mbc_growth_stages; + Uint sbmbc_threshold; + Uint sbmbc_size; #if HAVE_ERTS_MSEG ErtsMsegOpt_t mseg_opt; #endif @@ -269,6 +290,7 @@ struct Allctr_t_ { Uint min_block_size; /* Carriers */ + CarrierList_t sbmbc_list; CarrierList_t mbc_list; CarrierList_t sbc_list; @@ -277,15 +299,15 @@ struct Allctr_t_ { /* Callback functions (first 4 are mandatory) */ Block_t * (*get_free_block) (Allctr_t *, Uint, - Block_t *, Uint); - void (*link_free_block) (Allctr_t *, Block_t *); - void (*unlink_free_block) (Allctr_t *, Block_t *); + Block_t *, Uint, Uint32); + void (*link_free_block) (Allctr_t *, Block_t *, Uint32); + void (*unlink_free_block) (Allctr_t *, Block_t *, Uint32); Eterm (*info_options) (Allctr_t *, char *, int *, void *, Uint **, Uint *); Uint (*get_next_mbc_size) (Allctr_t *); - void (*creating_mbc) (Allctr_t *, Carrier_t *); - void (*destroying_mbc) (Allctr_t *, Carrier_t *); + void (*creating_mbc) (Allctr_t *, Carrier_t *, Uint32); + void (*destroying_mbc) (Allctr_t *, Carrier_t *, Uint32); void (*init_atoms) (void); #ifdef ERTS_ALLOC_UTIL_HARD_DEBUG @@ -312,6 +334,8 @@ struct Allctr_t_ { CallCounter_t this_alloc; CallCounter_t this_free; CallCounter_t this_realloc; + CallCounter_t sbmbc_alloc; + CallCounter_t sbmbc_free; CallCounter_t mseg_alloc; CallCounter_t mseg_dealloc; CallCounter_t mseg_realloc; @@ -322,6 +346,7 @@ struct Allctr_t_ { CarriersStats_t sbcs; CarriersStats_t mbcs; + CarriersStats_t sbmbcs; #ifdef DEBUG #ifdef USE_THREADS diff --git a/erts/emulator/beam/erl_ao_firstfit_alloc.c b/erts/emulator/beam/erl_ao_firstfit_alloc.c new file mode 100644 index 0000000000..002852cdad --- /dev/null +++ b/erts/emulator/beam/erl_ao_firstfit_alloc.c @@ -0,0 +1,972 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2003-2009. All Rights Reserved. + * + * The contents of this file are subject to the Erlang Public License, + * Version 1.1, (the "License"); you may not use this file except in + * compliance with the License. You should have received a copy of the + * Erlang Public License along with this software. If not, it can be + * retrieved online at http://www.erlang.org/. + * + * Software distributed under the License is distributed on an "AS IS" + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + * the License for the specific language governing rights and limitations + * under the License. + * + * %CopyrightEnd% + */ + + +/* + * Description: An "address order first fit" allocator + * based on a Red-Black (binary search) Tree. The search, + * insert, and delete operations are all O(log n) operations + * on a Red-Black Tree. + * Red-Black Trees are described in "Introduction to Algorithms", + * by Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Riverest. + * + * This module is a callback-module for erl_alloc_util.c + * + * Algorithm: The tree nodes are free-blocks ordered in address order. + * Every node also keeps the size of the largest block in its + * sub-tree ('max_size'). By that we can start from root and keep + * left (for low addresses) while dismissing entire sub-trees with + * too small blocks. + * + * Authors: Rickard Green/Sverker Eriksson + */ + + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif +#include "global.h" +#define GET_ERL_AOFF_ALLOC_IMPL +#include "erl_ao_firstfit_alloc.h" + +#ifdef DEBUG +#if 0 +#define HARD_DEBUG +#endif +#else +#undef HARD_DEBUG +#endif + +#define MIN_MBC_SZ (16*1024) +#define MIN_MBC_FIRST_FREE_SZ (4*1024) + +#define TREE_NODE_FLG (((Uint) 1) << 0) +#define RED_FLG (((Uint) 1) << 1) +#ifdef HARD_DEBUG +# define LEFT_VISITED_FLG (((Uint) 1) << 2) +# define RIGHT_VISITED_FLG (((Uint) 1) << 3) +#endif + +#define IS_RED(N) (((AOFF_RBTree_t *) (N)) \ + && ((AOFF_RBTree_t *) (N))->flags & RED_FLG) +#define IS_BLACK(N) (!IS_RED(((AOFF_RBTree_t *) (N)))) + +#define SET_RED(N) (((AOFF_RBTree_t *) (N))->flags |= RED_FLG) +#define SET_BLACK(N) (((AOFF_RBTree_t *) (N))->flags &= ~RED_FLG) + +#undef ASSERT +#define ASSERT ASSERT_EXPR + +#if 1 +#define RBT_ASSERT ASSERT +#else +#define RBT_ASSERT(x) +#endif + + +/* Types... */ +typedef struct AOFF_RBTree_t_ AOFF_RBTree_t; + +struct AOFF_RBTree_t_ { + Block_t hdr; + Uint flags; + AOFF_RBTree_t *parent; + AOFF_RBTree_t *left; + AOFF_RBTree_t *right; + Uint max_sz; /* of all blocks in this sub-tree */ +}; + +#ifdef HARD_DEBUG +static AOFF_RBTree_t * check_tree(AOFF_RBTree_t* root, Uint); +#endif + + +/* Calculate 'max_size' of tree node x by only looking at the direct children + * of x and x itself. + */ +static ERTS_INLINE Uint node_max_size(AOFF_RBTree_t *x) +{ + Uint sz = BLK_SZ(x); + if (x->left && x->left->max_sz > sz) { + sz = x->left->max_sz; + } + if (x->right && x->right->max_sz > sz) { + sz = x->right->max_sz; + } + return sz; +} + +/* Set new possibly lower 'max_size' of node and propagate change toward root +*/ +static ERTS_INLINE void lower_max_size(AOFF_RBTree_t *node, + AOFF_RBTree_t* stop_at) +{ + AOFF_RBTree_t* x = node; + Uint old_max = x->max_sz; + Uint new_max = node_max_size(x); + + if (new_max < old_max) { + x->max_sz = new_max; + while ((x=x->parent) != stop_at && x->max_sz == old_max) { + x->max_sz = node_max_size(x); + } + ASSERT(x == stop_at || x->max_sz > old_max); + } + else ASSERT(new_max == old_max); +} + + +/* Prototypes of callback functions */ +static Block_t* aoff_get_free_block(Allctr_t *, Uint, Block_t *, Uint, Uint32 flags); +static void aoff_link_free_block(Allctr_t *, Block_t*, Uint32 flags); +static void aoff_unlink_free_block(Allctr_t *allctr, Block_t *del, Uint32 flags); + +static Eterm info_options(Allctr_t *, char *, int *, void *, Uint **, Uint *); +static void init_atoms(void); + + + +#ifdef DEBUG + +/* Destroy all tree fields */ +#define DESTROY_TREE_NODE(N) \ + sys_memset((void *) (((Block_t *) (N)) + 1), \ + 0xff, \ + (sizeof(AOFF_RBTree_t) - sizeof(Block_t))) + +#else + +#define DESTROY_TREE_NODE(N) + +#endif + + +static int atoms_initialized = 0; + +void +erts_aoffalc_init(void) +{ + atoms_initialized = 0; +} + +Allctr_t * +erts_aoffalc_start(AOFFAllctr_t *alc, + AOFFAllctrInit_t* aoffinit, + AllctrInit_t *init) +{ + AOFFAllctr_t nulled_state = {{0}}; + /* {{0}} is used instead of {0}, in order to avoid (an incorrect) gcc + warning. gcc warns if {0} is used as initializer of a struct when + the first member is a struct (not if, for example, the third member + is a struct). */ + Allctr_t *allctr = (Allctr_t *) alc; + + sys_memcpy((void *) alc, (void *) &nulled_state, sizeof(AOFFAllctr_t)); + + allctr->mbc_header_size = sizeof(Carrier_t); + allctr->min_mbc_size = MIN_MBC_SZ; + allctr->min_mbc_first_free_size = MIN_MBC_FIRST_FREE_SZ; + allctr->min_block_size = sizeof(AOFF_RBTree_t); + + allctr->vsn_str = ERTS_ALC_AOFF_ALLOC_VSN_STR; + + + /* Callback functions */ + + allctr->get_free_block = aoff_get_free_block; + allctr->link_free_block = aoff_link_free_block; + allctr->unlink_free_block = aoff_unlink_free_block; + allctr->info_options = info_options; + + allctr->get_next_mbc_size = NULL; + allctr->creating_mbc = NULL; + allctr->destroying_mbc = NULL; + allctr->init_atoms = init_atoms; + +#ifdef ERTS_ALLOC_UTIL_HARD_DEBUG + allctr->check_block = NULL; + allctr->check_mbc = NULL; +#endif + + allctr->atoms_initialized = 0; + + if (!erts_alcu_start(allctr, init)) + return NULL; + + return allctr; +} + +/* + * Red-Black Tree operations needed + */ + +static ERTS_INLINE void +left_rotate(AOFF_RBTree_t **root, AOFF_RBTree_t *x) +{ + AOFF_RBTree_t *y = x->right; + x->right = y->left; + if (y->left) + y->left->parent = x; + y->parent = x->parent; + if (!y->parent) { + RBT_ASSERT(*root == x); + *root = y; + } + else if (x == x->parent->left) + x->parent->left = y; + else { + RBT_ASSERT(x == x->parent->right); + x->parent->right = y; + } + y->left = x; + x->parent = y; + + y->max_sz = x->max_sz; + x->max_sz = node_max_size(x); + ASSERT(y->max_sz >= x->max_sz); +} + +static ERTS_INLINE void +right_rotate(AOFF_RBTree_t **root, AOFF_RBTree_t *x) +{ + AOFF_RBTree_t *y = x->left; + x->left = y->right; + if (y->right) + y->right->parent = x; + y->parent = x->parent; + if (!y->parent) { + RBT_ASSERT(*root == x); + *root = y; + } + else if (x == x->parent->right) + x->parent->right = y; + else { + RBT_ASSERT(x == x->parent->left); + x->parent->left = y; + } + y->right = x; + x->parent = y; + y->max_sz = x->max_sz; + x->max_sz = node_max_size(x); + ASSERT(y->max_sz >= x->max_sz); +} + + +/* + * Replace node x with node y + * NOTE: block header of y is not changed + */ +static ERTS_INLINE void +replace(AOFF_RBTree_t **root, AOFF_RBTree_t *x, AOFF_RBTree_t *y) +{ + + if (!x->parent) { + RBT_ASSERT(*root == x); + *root = y; + } + else if (x == x->parent->left) + x->parent->left = y; + else { + RBT_ASSERT(x == x->parent->right); + x->parent->right = y; + } + if (x->left) { + RBT_ASSERT(x->left->parent == x); + x->left->parent = y; + } + if (x->right) { + RBT_ASSERT(x->right->parent == x); + x->right->parent = y; + } + + y->flags = x->flags; + y->parent = x->parent; + y->right = x->right; + y->left = x->left; + + y->max_sz = x->max_sz; + lower_max_size(y, NULL); + DESTROY_TREE_NODE(x); +} + +static void +tree_insert_fixup(AOFF_RBTree_t** root, AOFF_RBTree_t *blk) +{ + AOFF_RBTree_t *x = blk, *y; + + /* + * Rearrange the tree so that it satisfies the Red-Black Tree properties + */ + + RBT_ASSERT(x != *root && IS_RED(x->parent)); + do { + + /* + * x and its parent are both red. Move the red pair up the tree + * until we get to the root or until we can separate them. + */ + + RBT_ASSERT(IS_RED(x)); + RBT_ASSERT(IS_BLACK(x->parent->parent)); + RBT_ASSERT(x->parent->parent); + + if (x->parent == x->parent->parent->left) { + y = x->parent->parent->right; + if (IS_RED(y)) { + SET_BLACK(y); + x = x->parent; + SET_BLACK(x); + x = x->parent; + SET_RED(x); + } + else { + + if (x == x->parent->right) { + x = x->parent; + left_rotate(root, x); + } + + RBT_ASSERT(x == x->parent->parent->left->left); + RBT_ASSERT(IS_RED(x)); + RBT_ASSERT(IS_RED(x->parent)); + RBT_ASSERT(IS_BLACK(x->parent->parent)); + RBT_ASSERT(IS_BLACK(y)); + + SET_BLACK(x->parent); + SET_RED(x->parent->parent); + right_rotate(root, x->parent->parent); + + RBT_ASSERT(x == x->parent->left); + RBT_ASSERT(IS_RED(x)); + RBT_ASSERT(IS_RED(x->parent->right)); + RBT_ASSERT(IS_BLACK(x->parent)); + break; + } + } + else { + RBT_ASSERT(x->parent == x->parent->parent->right); + y = x->parent->parent->left; + if (IS_RED(y)) { + SET_BLACK(y); + x = x->parent; + SET_BLACK(x); + x = x->parent; + SET_RED(x); + } + else { + + if (x == x->parent->left) { + x = x->parent; + right_rotate(root, x); + } + + RBT_ASSERT(x == x->parent->parent->right->right); + RBT_ASSERT(IS_RED(x)); + RBT_ASSERT(IS_RED(x->parent)); + RBT_ASSERT(IS_BLACK(x->parent->parent)); + RBT_ASSERT(IS_BLACK(y)); + + SET_BLACK(x->parent); + SET_RED(x->parent->parent); + left_rotate(root, x->parent->parent); + + RBT_ASSERT(x == x->parent->right); + RBT_ASSERT(IS_RED(x)); + RBT_ASSERT(IS_RED(x->parent->left)); + RBT_ASSERT(IS_BLACK(x->parent)); + break; + } + } + } while (x != *root && IS_RED(x->parent)); + + SET_BLACK(*root); +} + +static void +aoff_unlink_free_block(Allctr_t *allctr, Block_t *del, Uint32 flags) +{ + AOFFAllctr_t *alc = (AOFFAllctr_t *) allctr; + AOFF_RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &alc->sbmbc_root : &alc->mbc_root); + Uint spliced_is_black; + AOFF_RBTree_t *x, *y, *z = (AOFF_RBTree_t *) del; + AOFF_RBTree_t null_x; /* null_x is used to get the fixup started when we + splice out a node without children. */ + + null_x.parent = NULL; + +#ifdef HARD_DEBUG + check_tree(*root, 0); +#endif + + /* Remove node from tree... */ + + /* Find node to splice out */ + if (!z->left || !z->right) + y = z; + else + /* Set y to z:s successor */ + for(y = z->right; y->left; y = y->left); + /* splice out y */ + x = y->left ? y->left : y->right; + spliced_is_black = IS_BLACK(y); + if (x) { + x->parent = y->parent; + } + else if (spliced_is_black) { + x = &null_x; + x->flags = 0; + SET_BLACK(x); + x->right = x->left = NULL; + x->max_sz = 0; + x->parent = y->parent; + y->left = x; + } + + if (!y->parent) { + RBT_ASSERT(*root == y); + *root = x; + } + else { + if (y == y->parent->left) { + y->parent->left = x; + } + else { + RBT_ASSERT(y == y->parent->right); + y->parent->right = x; + } + if (y->parent != z) { + lower_max_size(y->parent, (y==z ? NULL : z)); + } + } + if (y != z) { + /* We spliced out the successor of z; replace z by the successor */ + replace(root, z, y); + } + + if (spliced_is_black) { + /* We removed a black node which makes the resulting tree + violate the Red-Black Tree properties. Fixup tree... */ + + while (IS_BLACK(x) && x->parent) { + + /* + * x has an "extra black" which we move up the tree + * until we reach the root or until we can get rid of it. + * + * y is the sibbling of x + */ + + if (x == x->parent->left) { + y = x->parent->right; + RBT_ASSERT(y); + if (IS_RED(y)) { + RBT_ASSERT(y->right); + RBT_ASSERT(y->left); + SET_BLACK(y); + RBT_ASSERT(IS_BLACK(x->parent)); + SET_RED(x->parent); + left_rotate(root, x->parent); + y = x->parent->right; + } + RBT_ASSERT(y); + RBT_ASSERT(IS_BLACK(y)); + if (IS_BLACK(y->left) && IS_BLACK(y->right)) { + SET_RED(y); + x = x->parent; + } + else { + if (IS_BLACK(y->right)) { + SET_BLACK(y->left); + SET_RED(y); + right_rotate(root, y); + y = x->parent->right; + } + RBT_ASSERT(y); + if (IS_RED(x->parent)) { + + SET_BLACK(x->parent); + SET_RED(y); + } + RBT_ASSERT(y->right); + SET_BLACK(y->right); + left_rotate(root, x->parent); + x = *root; + break; + } + } + else { + RBT_ASSERT(x == x->parent->right); + y = x->parent->left; + RBT_ASSERT(y); + if (IS_RED(y)) { + RBT_ASSERT(y->right); + RBT_ASSERT(y->left); + SET_BLACK(y); + RBT_ASSERT(IS_BLACK(x->parent)); + SET_RED(x->parent); + right_rotate(root, x->parent); + y = x->parent->left; + } + RBT_ASSERT(y); + RBT_ASSERT(IS_BLACK(y)); + if (IS_BLACK(y->right) && IS_BLACK(y->left)) { + SET_RED(y); + x = x->parent; + } + else { + if (IS_BLACK(y->left)) { + SET_BLACK(y->right); + SET_RED(y); + left_rotate(root, y); + y = x->parent->left; + } + RBT_ASSERT(y); + if (IS_RED(x->parent)) { + SET_BLACK(x->parent); + SET_RED(y); + } + RBT_ASSERT(y->left); + SET_BLACK(y->left); + right_rotate(root, x->parent); + x = *root; + break; + } + } + } + SET_BLACK(x); + + if (null_x.parent) { + if (null_x.parent->left == &null_x) + null_x.parent->left = NULL; + else { + RBT_ASSERT(null_x.parent->right == &null_x); + null_x.parent->right = NULL; + } + RBT_ASSERT(!null_x.left); + RBT_ASSERT(!null_x.right); + } + else if (*root == &null_x) { + *root = NULL; + RBT_ASSERT(!null_x.left); + RBT_ASSERT(!null_x.right); + } + } + + DESTROY_TREE_NODE(del); + +#ifdef HARD_DEBUG + check_tree(*root, 0); +#endif +} + +static void +aoff_link_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) +{ + AOFFAllctr_t *alc = (AOFFAllctr_t *) allctr; + AOFF_RBTree_t *blk = (AOFF_RBTree_t *) block; + AOFF_RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &alc->sbmbc_root : &alc->mbc_root); + Uint blk_sz = BLK_SZ(blk); + +#ifdef HARD_DEBUG + check_tree(*root, 0); +#endif + + blk->flags = 0; + blk->left = NULL; + blk->right = NULL; + blk->max_sz = blk_sz; + + if (!*root) { + blk->parent = NULL; + SET_BLACK(blk); + *root = blk; + } + else { + AOFF_RBTree_t *x = *root; + while (1) { + if (x->max_sz < blk_sz) { + x->max_sz = blk_sz; + } + if (blk < x) { + if (!x->left) { + blk->parent = x; + x->left = blk; + break; + } + x = x->left; + } + else { + if (!x->right) { + blk->parent = x; + x->right = blk; + break; + } + x = x->right; + } + + } + + /* Insert block into size tree */ + RBT_ASSERT(blk->parent); + + SET_RED(blk); + if (IS_RED(blk->parent)) + tree_insert_fixup(root, blk); + } + +#ifdef HARD_DEBUG + check_tree(*root, 0); +#endif +} + +static Block_t * +aoff_get_free_block(Allctr_t *allctr, Uint size, + Block_t *cand_blk, Uint cand_size, Uint32 flags) +{ + AOFFAllctr_t *alc = (AOFFAllctr_t *) allctr; + AOFF_RBTree_t *x = ((flags & ERTS_ALCU_FLG_SBMBC) + ? alc->sbmbc_root : alc->mbc_root); + AOFF_RBTree_t *blk = NULL; +#ifdef HARD_DEBUG + AOFF_RBTree_t* dbg_blk = check_tree(x, size); +#endif + + ASSERT(!cand_blk || cand_size >= size); + + while (x) { + if (x->left && x->left->max_sz >= size) { + x = x->left; + } + else if (BLK_SZ(x) >= size) { + blk = x; + break; + } + else { + x = x->right; + } + } + +#ifdef HARD_DEBUG + ASSERT(blk == dbg_blk); +#endif + + if (!blk) + return NULL; + + if (cand_blk && cand_blk < &blk->hdr) { + return NULL; /* cand_blk was better */ + } + + aoff_unlink_free_block(allctr, (Block_t *) blk, flags); + + return (Block_t *) blk; +} + + +/* + * info_options() + */ + +static struct { + Eterm as; + Eterm aoff; +#ifdef DEBUG + Eterm end_of_atoms; +#endif +} am; + +static void ERTS_INLINE atom_init(Eterm *atom, char *name) +{ + *atom = am_atom_put(name, strlen(name)); +} +#define AM_INIT(AM) atom_init(&am.AM, #AM) + +static void +init_atoms(void) +{ +#ifdef DEBUG + Eterm *atom; +#endif + + if (atoms_initialized) + return; + +#ifdef DEBUG + for (atom = (Eterm *) &am; atom <= &am.end_of_atoms; atom++) { + *atom = THE_NON_VALUE; + } +#endif + AM_INIT(as); + AM_INIT(aoff); + +#ifdef DEBUG + for (atom = (Eterm *) &am; atom < &am.end_of_atoms; atom++) { + ASSERT(*atom != THE_NON_VALUE); + } +#endif + + atoms_initialized = 1; +} + + +#define bld_uint erts_bld_uint +#define bld_cons erts_bld_cons +#define bld_tuple erts_bld_tuple + +static ERTS_INLINE void +add_2tup(Uint **hpp, Uint *szp, Eterm *lp, Eterm el1, Eterm el2) +{ + *lp = bld_cons(hpp, szp, bld_tuple(hpp, szp, 2, el1, el2), *lp); +} + +static Eterm +info_options(Allctr_t *allctr, + char *prefix, + int *print_to_p, + void *print_to_arg, + Uint **hpp, + Uint *szp) +{ + Eterm res = THE_NON_VALUE; + + if (print_to_p) { + erts_print(*print_to_p, + print_to_arg, + "%sas: %s\n", + prefix, + "aoff"); + } + + if (hpp || szp) { + + if (!atoms_initialized) + erl_exit(1, "%s:%d: Internal error: Atoms not initialized", + __FILE__, __LINE__);; + + res = NIL; + add_2tup(hpp, szp, &res, am.as, am.aoff); + } + + return res; +} + + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * NOTE: erts_aoffalc_test() is only supposed to be used for testing. * + * * + * Keep alloc_SUITE_data/allocator_test.h updated if changes are made * + * to erts_aoffalc_test() * +\* */ + +unsigned long +erts_aoffalc_test(unsigned long op, unsigned long a1, unsigned long a2) +{ + switch (op) { + case 0x500: return (unsigned long) 0; /* IS_AOBF */ + case 0x501: return (unsigned long) ((AOFFAllctr_t *) a1)->mbc_root; + case 0x502: return (unsigned long) ((AOFF_RBTree_t *) a1)->parent; + case 0x503: return (unsigned long) ((AOFF_RBTree_t *) a1)->left; + case 0x504: return (unsigned long) ((AOFF_RBTree_t *) a1)->right; + case 0x506: return (unsigned long) IS_BLACK((AOFF_RBTree_t *) a1); + case 0x508: return (unsigned long) 1; /* IS_AOFF */ + case 0x509: return (unsigned long) ((AOFF_RBTree_t *) a1)->max_sz; + default: ASSERT(0); return ~((unsigned long) 0); + } +} + + +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\ + * Debug functions * +\* */ + + +#ifdef HARD_DEBUG + +#define IS_LEFT_VISITED(FB) ((FB)->flags & LEFT_VISITED_FLG) +#define IS_RIGHT_VISITED(FB) ((FB)->flags & RIGHT_VISITED_FLG) + +#define SET_LEFT_VISITED(FB) ((FB)->flags |= LEFT_VISITED_FLG) +#define SET_RIGHT_VISITED(FB) ((FB)->flags |= RIGHT_VISITED_FLG) + +#define UNSET_LEFT_VISITED(FB) ((FB)->flags &= ~LEFT_VISITED_FLG) +#define UNSET_RIGHT_VISITED(FB) ((FB)->flags &= ~RIGHT_VISITED_FLG) + + +#if 0 +# define PRINT_TREE +#else +# undef PRINT_TREE +#endif + +#ifdef PRINT_TREE +static void print_tree(AOFF_RBTree_t*); +#endif + +/* + * Checks that the order between parent and children are correct, + * and that the Red-Black Tree properies are satisfied. if size > 0, + * check_tree() returns the node that satisfies "address order first fit" + * + * The Red-Black Tree properies are: + * 1. Every node is either red or black. + * 2. Every leaf (NIL) is black. + * 3. If a node is red, then both its children are black. + * 4. Every simple path from a node to a descendant leaf + * contains the same number of black nodes. + * + * + own.max_size == MAX(own.size, left.max_size, right.max_size) + */ + +static AOFF_RBTree_t * +check_tree(AOFF_RBTree_t* root, Uint size) +{ + AOFF_RBTree_t *res = NULL; + Sint blacks; + Sint curr_blacks; + AOFF_RBTree_t *x; + +#ifdef PRINT_TREE + print_tree(root); +#endif + + if (!root) + return res; + + x = root; + ASSERT(IS_BLACK(x)); + ASSERT(!x->parent); + curr_blacks = 1; + blacks = -1; + + while (x) { + if (!IS_LEFT_VISITED(x)) { + SET_LEFT_VISITED(x); + if (x->left) { + x = x->left; + if (IS_BLACK(x)) + curr_blacks++; + continue; + } + else { + if (blacks < 0) + blacks = curr_blacks; + ASSERT(blacks == curr_blacks); + } + } + + if (!IS_RIGHT_VISITED(x)) { + SET_RIGHT_VISITED(x); + if (x->right) { + x = x->right; + if (IS_BLACK(x)) + curr_blacks++; + continue; + } + else { + if (blacks < 0) + blacks = curr_blacks; + ASSERT(blacks == curr_blacks); + } + } + + + if (IS_RED(x)) { + ASSERT(IS_BLACK(x->right)); + ASSERT(IS_BLACK(x->left)); + } + + ASSERT(x->parent || x == root); + + if (x->left) { + ASSERT(x->left->parent == x); + ASSERT(x->left < x); + ASSERT(x->left->max_sz <= x->max_sz); + } + + if (x->right) { + ASSERT(x->right->parent == x); + ASSERT(x->right > x); + ASSERT(x->right->max_sz <= x->max_sz); + } + ASSERT(x->max_sz >= BLK_SZ(x)); + ASSERT(x->max_sz == BLK_SZ(x) + || x->max_sz == (x->left ? x->left->max_sz : 0) + || x->max_sz == (x->right ? x->right->max_sz : 0)); + + if (size && BLK_SZ(x) >= size) { + if (!res || x < res) { + res = x; + } + } + + UNSET_LEFT_VISITED(x); + UNSET_RIGHT_VISITED(x); + if (IS_BLACK(x)) + curr_blacks--; + x = x->parent; + + } + + ASSERT(curr_blacks == 0); + + UNSET_LEFT_VISITED(root); + UNSET_RIGHT_VISITED(root); + + return res; + +} + + +#ifdef PRINT_TREE +#define INDENT_STEP 2 + +#include <stdio.h> + +static void +print_tree_aux(AOFF_RBTree_t *x, int indent) +{ + int i; + + if (x) { + print_tree_aux(x->right, indent + INDENT_STEP); + for (i = 0; i < indent; i++) { + putc(' ', stderr); + } + fprintf(stderr, "%s: sz=%lu addr=0x%lx max_size=%lu\r\n", + IS_BLACK(x) ? "BLACK" : "RED", + BLK_SZ(x), (Uint)x, x->max_sz); + print_tree_aux(x->left, indent + INDENT_STEP); + } +} + + +static void +print_tree(AOFF_RBTree_t* root) +{ + fprintf(stderr, " --- AOFF tree begin ---\r\n"); + print_tree_aux(root, 0); + fprintf(stderr, " --- AOFF tree end ---\r\n"); +} + +#endif /* PRINT_TREE */ + +#endif /* HARD_DEBUG */ + diff --git a/erts/emulator/beam/erl_ao_firstfit_alloc.h b/erts/emulator/beam/erl_ao_firstfit_alloc.h new file mode 100644 index 0000000000..0bf0ec8cee --- /dev/null +++ b/erts/emulator/beam/erl_ao_firstfit_alloc.h @@ -0,0 +1,60 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2003-2009. All Rights Reserved. + * + * The contents of this file are subject to the Erlang Public License, + * Version 1.1, (the "License"); you may not use this file except in + * compliance with the License. You should have received a copy of the + * Erlang Public License along with this software. If not, it can be + * retrieved online at http://www.erlang.org/. + * + * Software distributed under the License is distributed on an "AS IS" + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + * the License for the specific language governing rights and limitations + * under the License. + * + * %CopyrightEnd% + */ + + +#ifndef ERL_AO_FIRSTFIT_ALLOC__ +#define ERL_AO_FIRSTFIT_ALLOC__ + +#include "erl_alloc_util.h" + +#define ERTS_ALC_AOFF_ALLOC_VSN_STR "0.9" + +typedef struct AOFFAllctr_t_ AOFFAllctr_t; + +typedef struct { + int dummy; +} AOFFAllctrInit_t; + +#define ERTS_DEFAULT_AOFF_ALLCTR_INIT {0/*dummy*/} + +void erts_aoffalc_init(void); +Allctr_t *erts_aoffalc_start(AOFFAllctr_t *, AOFFAllctrInit_t*, AllctrInit_t *); + +#endif /* #ifndef ERL_AO_FIRSTFIT_ALLOC__ */ + + + +#if defined(GET_ERL_AOFF_ALLOC_IMPL) && !defined(ERL_AOFF_ALLOC_IMPL__) +#define ERL_AOFF_ALLOC_IMPL__ + +#define GET_ERL_ALLOC_UTIL_IMPL +#include "erl_alloc_util.h" + + +struct AOFFAllctr_t_ { + Allctr_t allctr; /* Has to be first! */ + + struct AOFF_RBTree_t_* mbc_root; + struct AOFF_RBTree_t_* sbmbc_root; +}; + +unsigned long erts_aoffalc_test(unsigned long, unsigned long, unsigned long); + +#endif /* #if defined(GET_ERL_AOFF_ALLOC_IMPL) + && !defined(ERL_AOFF_ALLOC_IMPL__) */ diff --git a/erts/emulator/beam/erl_bestfit_alloc.c b/erts/emulator/beam/erl_bestfit_alloc.c index 3035e5df16..5e3032ddaa 100644 --- a/erts/emulator/beam/erl_bestfit_alloc.c +++ b/erts/emulator/beam/erl_bestfit_alloc.c @@ -84,24 +84,24 @@ #ifdef HARD_DEBUG -static RBTree_t * check_tree(BFAllctr_t *, Uint); +static RBTree_t * check_tree(RBTree_t, int, Uint); #endif -static void tree_delete(Allctr_t *allctr, Block_t *del); +static void tree_delete(Allctr_t *allctr, Block_t *del, Uint32 flags); /* Prototypes of callback functions */ /* "address order best fit" specific callback functions */ static Block_t * aobf_get_free_block (Allctr_t *, Uint, - Block_t *, Uint); -static void aobf_link_free_block (Allctr_t *, Block_t *); + Block_t *, Uint, Uint32); +static void aobf_link_free_block (Allctr_t *, Block_t *, Uint32); #define aobf_unlink_free_block tree_delete /* "best fit" specific callback functions */ static Block_t * bf_get_free_block (Allctr_t *, Uint, - Block_t *, Uint); -static void bf_link_free_block (Allctr_t *, Block_t *); -static ERTS_INLINE void bf_unlink_free_block (Allctr_t *, Block_t *); + Block_t *, Uint, Uint32); +static void bf_link_free_block (Allctr_t *, Block_t *, Uint32); +static ERTS_INLINE void bf_unlink_free_block (Allctr_t *, Block_t *, Uint32); static Eterm info_options (Allctr_t *, char *, int *, @@ -303,7 +303,7 @@ replace(RBTree_t **root, RBTree_t *x, RBTree_t *y) } static void -tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) +tree_insert_fixup(RBTree_t **root, RBTree_t *blk) { RBTree_t *x = blk, *y; @@ -311,7 +311,7 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) * Rearrange the tree so that it satisfies the Red-Black Tree properties */ - RBT_ASSERT(x != bfallctr->root && IS_RED(x->parent)); + RBT_ASSERT(x != *root && IS_RED(x->parent)); do { /* @@ -336,7 +336,7 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) if (x == x->parent->right) { x = x->parent; - left_rotate(&bfallctr->root, x); + left_rotate(root, x); } RBT_ASSERT(x == x->parent->parent->left->left); @@ -347,7 +347,7 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) SET_BLACK(x->parent); SET_RED(x->parent->parent); - right_rotate(&bfallctr->root, x->parent->parent); + right_rotate(root, x->parent->parent); RBT_ASSERT(x == x->parent->left); RBT_ASSERT(IS_RED(x)); @@ -370,7 +370,7 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) if (x == x->parent->left) { x = x->parent; - right_rotate(&bfallctr->root, x); + right_rotate(root, x); } RBT_ASSERT(x == x->parent->parent->right->right); @@ -381,7 +381,7 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) SET_BLACK(x->parent); SET_RED(x->parent->parent); - left_rotate(&bfallctr->root, x->parent->parent); + left_rotate(root, x->parent->parent); RBT_ASSERT(x == x->parent->right); RBT_ASSERT(IS_RED(x)); @@ -390,9 +390,9 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) break; } } - } while (x != bfallctr->root && IS_RED(x->parent)); + } while (x != *root && IS_RED(x->parent)); - SET_BLACK(bfallctr->root); + SET_BLACK(*root); } @@ -402,18 +402,22 @@ tree_insert_fixup(BFAllctr_t *bfallctr, RBTree_t *blk) * callback function in the address order case. */ static void -tree_delete(Allctr_t *allctr, Block_t *del) +tree_delete(Allctr_t *allctr, Block_t *del, Uint32 flags) { BFAllctr_t *bfallctr = (BFAllctr_t *) allctr; Uint spliced_is_black; + RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &bfallctr->sbmbc_root + : &bfallctr->mbc_root); RBTree_t *x, *y, *z = (RBTree_t *) del; RBTree_t null_x; /* null_x is used to get the fixup started when we splice out a node without children. */ null_x.parent = NULL; + #ifdef HARD_DEBUG - check_tree(bfallctr, 0); + check_tree(*root, bfallctr->address_order, 0); #endif /* Remove node from tree... */ @@ -440,8 +444,8 @@ tree_delete(Allctr_t *allctr, Block_t *del) } if (!y->parent) { - RBT_ASSERT(bfallctr->root == y); - bfallctr->root = x; + RBT_ASSERT(*root == y); + *root = x; } else if (y == y->parent->left) y->parent->left = x; @@ -451,7 +455,7 @@ tree_delete(Allctr_t *allctr, Block_t *del) } if (y != z) { /* We spliced out the successor of z; replace z by the successor */ - replace(&bfallctr->root, z, y); + replace(root, z, y); } if (spliced_is_black) { @@ -476,7 +480,7 @@ tree_delete(Allctr_t *allctr, Block_t *del) SET_BLACK(y); RBT_ASSERT(IS_BLACK(x->parent)); SET_RED(x->parent); - left_rotate(&bfallctr->root, x->parent); + left_rotate(root, x->parent); y = x->parent->right; } RBT_ASSERT(y); @@ -489,7 +493,7 @@ tree_delete(Allctr_t *allctr, Block_t *del) if (IS_BLACK(y->right)) { SET_BLACK(y->left); SET_RED(y); - right_rotate(&bfallctr->root, y); + right_rotate(root, y); y = x->parent->right; } RBT_ASSERT(y); @@ -500,8 +504,8 @@ tree_delete(Allctr_t *allctr, Block_t *del) } RBT_ASSERT(y->right); SET_BLACK(y->right); - left_rotate(&bfallctr->root, x->parent); - x = bfallctr->root; + left_rotate(root, x->parent); + x = *root; break; } } @@ -515,7 +519,7 @@ tree_delete(Allctr_t *allctr, Block_t *del) SET_BLACK(y); RBT_ASSERT(IS_BLACK(x->parent)); SET_RED(x->parent); - right_rotate(&bfallctr->root, x->parent); + right_rotate(root, x->parent); y = x->parent->left; } RBT_ASSERT(y); @@ -528,7 +532,7 @@ tree_delete(Allctr_t *allctr, Block_t *del) if (IS_BLACK(y->left)) { SET_BLACK(y->right); SET_RED(y); - left_rotate(&bfallctr->root, y); + left_rotate(root, y); y = x->parent->left; } RBT_ASSERT(y); @@ -538,8 +542,8 @@ tree_delete(Allctr_t *allctr, Block_t *del) } RBT_ASSERT(y->left); SET_BLACK(y->left); - right_rotate(&bfallctr->root, x->parent); - x = bfallctr->root; + right_rotate(root, x->parent); + x = *root; break; } } @@ -556,8 +560,8 @@ tree_delete(Allctr_t *allctr, Block_t *del) RBT_ASSERT(!null_x.left); RBT_ASSERT(!null_x.right); } - else if (bfallctr->root == &null_x) { - bfallctr->root = NULL; + else if (*root == &null_x) { + *root = NULL; RBT_ASSERT(!null_x.left); RBT_ASSERT(!null_x.right); } @@ -567,7 +571,7 @@ tree_delete(Allctr_t *allctr, Block_t *del) DESTROY_TREE_NODE(del); #ifdef HARD_DEBUG - check_tree(bfallctr, 0); + check_tree(root, bfallctr->address_order, 0); #endif } @@ -577,23 +581,28 @@ tree_delete(Allctr_t *allctr, Block_t *del) \* */ static void -aobf_link_free_block(Allctr_t *allctr, Block_t *block) +aobf_link_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { BFAllctr_t *bfallctr = (BFAllctr_t *) allctr; + RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &bfallctr->sbmbc_root + : &bfallctr->mbc_root); RBTree_t *blk = (RBTree_t *) block; Uint blk_sz = BLK_SZ(blk); + + blk->flags = 0; blk->left = NULL; blk->right = NULL; - if (!bfallctr->root) { + if (!*root) { blk->parent = NULL; SET_BLACK(blk); - bfallctr->root = blk; + *root = blk; } else { - RBTree_t *x = bfallctr->root; + RBTree_t *x = *root; while (1) { Uint size; @@ -623,28 +632,32 @@ aobf_link_free_block(Allctr_t *allctr, Block_t *block) SET_RED(blk); if (IS_RED(blk->parent)) - tree_insert_fixup(bfallctr, blk); + tree_insert_fixup(root, blk); } #ifdef HARD_DEBUG - check_tree(bfallctr, 0); + check_tree(root, 1, 0); #endif } #if 0 /* tree_delete() is directly used instead */ static void -aobf_unlink_free_block(Allctr_t *allctr, Block_t *block) +aobf_unlink_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { - tree_delete(allctr, block); + tree_delete(allctr, block, flags); } #endif static Block_t * aobf_get_free_block(Allctr_t *allctr, Uint size, - Block_t *cand_blk, Uint cand_size) + Block_t *cand_blk, Uint cand_size, + Uint32 flags) { BFAllctr_t *bfallctr = (BFAllctr_t *) allctr; - RBTree_t *x = bfallctr->root; + RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &bfallctr->sbmbc_root + : &bfallctr->mbc_root); + RBTree_t *x = *root; RBTree_t *blk = NULL; Uint blk_sz; @@ -665,7 +678,7 @@ aobf_get_free_block(Allctr_t *allctr, Uint size, return NULL; #ifdef HARD_DEBUG - ASSERT(blk == check_tree(bfallctr, size)); + ASSERT(blk == check_tree(root, 1, size)); #endif if (cand_blk) { @@ -676,7 +689,7 @@ aobf_get_free_block(Allctr_t *allctr, Uint size, return NULL; /* cand_blk was better */ } - aobf_unlink_free_block(allctr, (Block_t *) blk); + aobf_unlink_free_block(allctr, (Block_t *) blk, flags); return (Block_t *) blk; } @@ -687,9 +700,12 @@ aobf_get_free_block(Allctr_t *allctr, Uint size, \* */ static void -bf_link_free_block(Allctr_t *allctr, Block_t *block) +bf_link_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { BFAllctr_t *bfallctr = (BFAllctr_t *) allctr; + RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &bfallctr->sbmbc_root + : &bfallctr->mbc_root); RBTree_t *blk = (RBTree_t *) block; Uint blk_sz = BLK_SZ(blk); @@ -700,13 +716,13 @@ bf_link_free_block(Allctr_t *allctr, Block_t *block) blk->left = NULL; blk->right = NULL; - if (!bfallctr->root) { + if (!*root) { blk->parent = NULL; SET_BLACK(blk); - bfallctr->root = blk; + *root = blk; } else { - RBTree_t *x = bfallctr->root; + RBTree_t *x = *root; while (1) { Uint size; @@ -745,7 +761,7 @@ bf_link_free_block(Allctr_t *allctr, Block_t *block) SET_RED(blk); if (IS_RED(blk->parent)) - tree_insert_fixup(bfallctr, blk); + tree_insert_fixup(root, blk); } @@ -753,14 +769,17 @@ bf_link_free_block(Allctr_t *allctr, Block_t *block) LIST_NEXT(blk) = NULL; #ifdef HARD_DEBUG - check_tree(bfallctr, 0); + check_tree(root, 0, 0); #endif } static ERTS_INLINE void -bf_unlink_free_block(Allctr_t *allctr, Block_t *block) +bf_unlink_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { BFAllctr_t *bfallctr = (BFAllctr_t *) allctr; + RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &bfallctr->sbmbc_root + : &bfallctr->mbc_root); RBTree_t *x = (RBTree_t *) block; if (IS_LIST_ELEM(x)) { @@ -778,9 +797,9 @@ bf_unlink_free_block(Allctr_t *allctr, Block_t *block) ASSERT(IS_LIST_ELEM(LIST_NEXT(x))); #ifdef HARD_DEBUG - check_tree(bfallctr, 0); + check_tree(root, 0, 0); #endif - replace(&bfallctr->root, x, LIST_NEXT(x)); + replace(root, x, LIST_NEXT(x)); #ifdef HARD_DEBUG check_tree(bfallctr, 0); @@ -788,7 +807,7 @@ bf_unlink_free_block(Allctr_t *allctr, Block_t *block) } else { /* Remove from tree */ - tree_delete(allctr, block); + tree_delete(allctr, block, flags); } DESTROY_LIST_ELEM(x); @@ -797,10 +816,14 @@ bf_unlink_free_block(Allctr_t *allctr, Block_t *block) static Block_t * bf_get_free_block(Allctr_t *allctr, Uint size, - Block_t *cand_blk, Uint cand_size) + Block_t *cand_blk, Uint cand_size, + Uint32 flags) { BFAllctr_t *bfallctr = (BFAllctr_t *) allctr; - RBTree_t *x = bfallctr->root; + RBTree_t **root = ((flags & ERTS_ALCU_FLG_SBMBC) + ? &bfallctr->sbmbc_root + : &bfallctr->mbc_root); + RBTree_t *x = *root; RBTree_t *blk = NULL; Uint blk_sz; @@ -827,7 +850,7 @@ bf_get_free_block(Allctr_t *allctr, Uint size, #ifdef HARD_DEBUG { - RBTree_t *ct_blk = check_tree(bfallctr, size); + RBTree_t *ct_blk = check_tree(root, 0, size); ASSERT(BLK_SZ(ct_blk) == BLK_SZ(blk)); } #endif @@ -839,7 +862,7 @@ bf_get_free_block(Allctr_t *allctr, Uint size, the tree node */ blk = LIST_NEXT(blk) ? LIST_NEXT(blk) : blk; - bf_unlink_free_block(allctr, (Block_t *) blk); + bf_unlink_free_block(allctr, (Block_t *) blk, flags); return (Block_t *) blk; } @@ -949,13 +972,14 @@ erts_bfalc_test(unsigned long op, unsigned long a1, unsigned long a2) { switch (op) { case 0x200: return (unsigned long) ((BFAllctr_t *) a1)->address_order; - case 0x201: return (unsigned long) ((BFAllctr_t *) a1)->root; + case 0x201: return (unsigned long) ((BFAllctr_t *) a1)->mbc_root; case 0x202: return (unsigned long) ((RBTree_t *) a1)->parent; case 0x203: return (unsigned long) ((RBTree_t *) a1)->left; case 0x204: return (unsigned long) ((RBTree_t *) a1)->right; case 0x205: return (unsigned long) ((RBTreeList_t *) a1)->next; case 0x206: return (unsigned long) IS_BLACK((RBTree_t *) a1); case 0x207: return (unsigned long) IS_TREE_NODE((RBTree_t *) a1); + case 0x208: return (unsigned long) 0; /* IS_AOFF */ default: ASSERT(0); return ~((unsigned long) 0); } } @@ -985,7 +1009,7 @@ erts_bfalc_test(unsigned long op, unsigned long a1, unsigned long a2) #endif #ifdef PRINT_TREE -static void print_tree(BFAllctr_t *); +static void print_tree(RBTree_t *, int); #endif /* @@ -1003,7 +1027,7 @@ static void print_tree(BFAllctr_t *); */ static RBTree_t * -check_tree(BFAllctr_t *bfallctr, Uint size) +check_tree(RBTree_t *root, int ao, Uint size) { RBTree_t *res = NULL; Sint blacks; @@ -1011,13 +1035,13 @@ check_tree(BFAllctr_t *bfallctr, Uint size) RBTree_t *x; #ifdef PRINT_TREE - print_tree(bfallctr); + print_tree(root, ao); #endif - if (!bfallctr->root) + if (!root) return res; - x = bfallctr->root; + x = root; ASSERT(IS_BLACK(x)); ASSERT(!x->parent); curr_blacks = 1; @@ -1060,11 +1084,11 @@ check_tree(BFAllctr_t *bfallctr, Uint size) ASSERT(IS_BLACK(x->left)); } - ASSERT(x->parent || x == bfallctr->root); + ASSERT(x->parent || x == root); if (x->left) { ASSERT(x->left->parent == x); - if (bfallctr->address_order) { + if (ao) { ASSERT(BLK_SZ(x->left) < BLK_SZ(x) || (BLK_SZ(x->left) == BLK_SZ(x) && x->left < x)); } @@ -1076,7 +1100,7 @@ check_tree(BFAllctr_t *bfallctr, Uint size) if (x->right) { ASSERT(x->right->parent == x); - if (bfallctr->address_order) { + if (ao) { ASSERT(BLK_SZ(x->right) > BLK_SZ(x) || (BLK_SZ(x->right) == BLK_SZ(x) && x->right > x)); } @@ -1087,7 +1111,7 @@ check_tree(BFAllctr_t *bfallctr, Uint size) } if (size && BLK_SZ(x) >= size) { - if (bfallctr->address_order) { + if (ao) { if (!res || BLK_SZ(x) < BLK_SZ(res) || (BLK_SZ(x) == BLK_SZ(res) && x < res)) @@ -1109,8 +1133,8 @@ check_tree(BFAllctr_t *bfallctr, Uint size) ASSERT(curr_blacks == 0); - UNSET_LEFT_VISITED(bfallctr->root); - UNSET_RIGHT_VISITED(bfallctr->root); + UNSET_LEFT_VISITED(root); + UNSET_RIGHT_VISITED(root); return res; @@ -1148,11 +1172,11 @@ print_tree_aux(RBTree_t *x, int indent) static void -print_tree(BFAllctr_t *bfallctr) +print_tree(RBTree_t *root, int ao) { - char *type = bfallctr->address_order ? "Size-Adress" : "Size"; + char *type = ao ? "Size-Adress" : "Size"; fprintf(stderr, " --- %s tree begin ---\r\n", type); - print_tree_aux(bfallctr->root, 0); + print_tree_aux(root, 0); fprintf(stderr, " --- %s tree end ---\r\n", type); } diff --git a/erts/emulator/beam/erl_bestfit_alloc.h b/erts/emulator/beam/erl_bestfit_alloc.h index cb35e21e57..faa2d9742e 100644 --- a/erts/emulator/beam/erl_bestfit_alloc.h +++ b/erts/emulator/beam/erl_bestfit_alloc.h @@ -54,7 +54,8 @@ typedef struct RBTree_t_ RBTree_t; struct BFAllctr_t_ { Allctr_t allctr; /* Has to be first! */ - RBTree_t * root; + RBTree_t * mbc_root; + RBTree_t * sbmbc_root; int address_order; }; diff --git a/erts/emulator/beam/erl_bits.h b/erts/emulator/beam/erl_bits.h index 0f67733fa4..3309ea706b 100644 --- a/erts/emulator/beam/erl_bits.h +++ b/erts/emulator/beam/erl_bits.h @@ -150,7 +150,7 @@ void erts_bits_destroy_state(ERL_BITS_PROTO_0); * NBYTES(x) returns the number of bytes needed to store x bits. */ -#define NBYTES(x) (((x) + 7) >> 3) +#define NBYTES(x) (((Uint64)(x) + (Uint64) 7) >> 3) #define BYTE_OFFSET(ofs) ((Uint) (ofs) >> 3) #define BIT_OFFSET(ofs) ((ofs) & 7) diff --git a/erts/emulator/beam/erl_db.c b/erts/emulator/beam/erl_db.c index e0a6aa05c6..eb50f56502 100644 --- a/erts/emulator/beam/erl_db.c +++ b/erts/emulator/beam/erl_db.c @@ -338,13 +338,13 @@ static ERTS_INLINE void db_unlock(DbTable* tb, db_lock_kind_t kind) ASSERT(tb != meta_pid_to_tab && tb != meta_pid_to_fixed_tab); if (tb->common.type & DB_FINE_LOCKED) { - if (tb->common.is_thread_safe) { - ASSERT(kind == LCK_WRITE); + if (kind == LCK_WRITE) { + ASSERT(tb->common.is_thread_safe); tb->common.is_thread_safe = 0; erts_smp_rwmtx_rwunlock(&tb->common.rwlock); } else { - ASSERT(kind != LCK_WRITE); + ASSERT(!tb->common.is_thread_safe); erts_smp_rwmtx_runlock(&tb->common.rwlock); } } @@ -543,9 +543,9 @@ static int remove_named_tab(DbTable *tb, int have_lock) * We keep our increased refc over this op in order to * prevent the table from disapearing. */ - erts_smp_rwmtx_rwunlock(&tb->common.rwlock); + db_unlock(tb, LCK_WRITE); erts_smp_rwmtx_rwlock(rwlock); - erts_smp_rwmtx_rwlock(&tb->common.rwlock); + db_lock(tb, LCK_WRITE); } #endif @@ -1650,24 +1650,6 @@ BIF_RETTYPE ets_delete_1(BIF_ALIST_1) tb->common.status &= ~(DB_PROTECTED|DB_PUBLIC|DB_PRIVATE); tb->common.status |= DB_DELETE; - mmtl = get_meta_main_tab_lock(tb->common.slot); -#ifdef ERTS_SMP - if (erts_smp_rwmtx_tryrwlock(mmtl) == EBUSY) { - /* - * We keep our increased refc over this op in order to - * prevent the table from disapearing. - */ - erts_smp_rwmtx_rwunlock(&tb->common.rwlock); - erts_smp_rwmtx_rwlock(mmtl); - erts_smp_rwmtx_rwlock(&tb->common.rwlock); - } -#endif - /* We must keep the slot, to be found by db_proc_dead() if process dies */ - MARK_SLOT_DEAD(tb->common.slot); - erts_smp_rwmtx_rwunlock(mmtl); - if (is_atom(tb->common.id)) - remove_named_tab(tb, 0); - if (tb->common.owner != BIF_P->id) { DeclareTmpHeap(meta_tuple,3,BIF_P); @@ -1691,6 +1673,25 @@ BIF_RETTYPE ets_delete_1(BIF_ALIST_1) db_meta_unlock(meta_pid_to_tab, LCK_WRITE_REC); UnUseTmpHeap(3,BIF_P); } + + mmtl = get_meta_main_tab_lock(tb->common.slot); +#ifdef ERTS_SMP + if (erts_smp_rwmtx_tryrwlock(mmtl) == EBUSY) { + /* + * We keep our increased refc over this op in order to + * prevent the table from disapearing. + */ + db_unlock(tb, LCK_WRITE); + erts_smp_rwmtx_rwlock(mmtl); + db_lock(tb, LCK_WRITE); + } +#endif + /* We must keep the slot, to be found by db_proc_dead() if process dies */ + MARK_SLOT_DEAD(tb->common.slot); + erts_smp_rwmtx_rwunlock(mmtl); + if (is_atom(tb->common.id)) + remove_named_tab(tb, 0); + /* disable inheritance */ free_heir_data(tb); tb->common.heir = am_none; diff --git a/erts/emulator/beam/erl_db_util.c b/erts/emulator/beam/erl_db_util.c index c3b074f782..e5be1f253a 100644 --- a/erts/emulator/beam/erl_db_util.c +++ b/erts/emulator/beam/erl_db_util.c @@ -1731,6 +1731,7 @@ Eterm db_prog_match(Process *c_p, Binary *bprog, #define BEGIN_ATOMIC_TRACE(p) \ do { \ if (! atomic_trace) { \ + erts_refc_inc(&bprog->refc, 2); \ erts_smp_proc_unlock((p), ERTS_PROC_LOCK_MAIN); \ erts_smp_block_system(0); \ atomic_trace = !0; \ @@ -1741,6 +1742,9 @@ Eterm db_prog_match(Process *c_p, Binary *bprog, if (atomic_trace) { \ erts_smp_release_system(); \ erts_smp_proc_lock((p), ERTS_PROC_LOCK_MAIN); \ + if (erts_refc_dectest(&bprog->refc, 0) == 0) {\ + erts_bin_free(bprog); \ + } \ atomic_trace = 0; \ } \ } while (0) diff --git a/erts/emulator/beam/erl_goodfit_alloc.c b/erts/emulator/beam/erl_goodfit_alloc.c index 76b206d76f..1cc508ac5a 100644 --- a/erts/emulator/beam/erl_goodfit_alloc.c +++ b/erts/emulator/beam/erl_goodfit_alloc.c @@ -163,10 +163,10 @@ BKT_MIN_SZ(GFAllctr_t *gfallctr, int ix) /* Prototypes of callback functions */ static Block_t * get_free_block (Allctr_t *, Uint, - Block_t *, Uint); -static void link_free_block (Allctr_t *, Block_t *); -static void unlink_free_block (Allctr_t *, Block_t *); -static void update_last_aux_mbc (Allctr_t *, Carrier_t *); + Block_t *, Uint, Uint32); +static void link_free_block (Allctr_t *, Block_t *, Uint32); +static void unlink_free_block (Allctr_t *, Block_t *, Uint32); +static void update_last_aux_mbc (Allctr_t *, Carrier_t *, Uint32); static Eterm info_options (Allctr_t *, char *, int *, void *, Uint **, Uint *); static void init_atoms (void); @@ -197,6 +197,8 @@ erts_gfalc_start(GFAllctr_t *gfallctr, is a struct). */ Allctr_t *allctr = (Allctr_t *) gfallctr; + init->sbmbct = 0; /* Small mbc not yet supported by goodfit */ + sys_memcpy((void *) gfallctr, (void *) &nulled_state, sizeof(GFAllctr_t)); allctr->mbc_header_size = sizeof(Carrier_t); @@ -379,7 +381,7 @@ search_bucket(Allctr_t *allctr, int ix, Uint size) static Block_t * get_free_block(Allctr_t *allctr, Uint size, - Block_t *cand_blk, Uint cand_size) + Block_t *cand_blk, Uint cand_size, Uint32 flags) { GFAllctr_t *gfallctr = (GFAllctr_t *) allctr; int unsafe_bi, min_bi; @@ -398,7 +400,7 @@ get_free_block(Allctr_t *allctr, Uint size, if (blk) { if (cand_blk && cand_size <= BLK_SZ(blk)) return NULL; /* cand_blk was better */ - unlink_free_block(allctr, blk); + unlink_free_block(allctr, blk, flags); return blk; } if (min_bi < NO_OF_BKTS - 1) { @@ -418,14 +420,14 @@ get_free_block(Allctr_t *allctr, Uint size, ASSERT(blk); if (cand_blk && cand_size <= BLK_SZ(blk)) return NULL; /* cand_blk was better */ - unlink_free_block(allctr, blk); + unlink_free_block(allctr, blk, flags); return blk; } static void -link_free_block(Allctr_t *allctr, Block_t *block) +link_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { GFAllctr_t *gfallctr = (GFAllctr_t *) allctr; GFFreeBlock_t *blk = (GFFreeBlock_t *) block; @@ -446,7 +448,7 @@ link_free_block(Allctr_t *allctr, Block_t *block) } static void -unlink_free_block(Allctr_t *allctr, Block_t *block) +unlink_free_block(Allctr_t *allctr, Block_t *block, Uint32 flags) { GFAllctr_t *gfallctr = (GFAllctr_t *) allctr; GFFreeBlock_t *blk = (GFFreeBlock_t *) block; @@ -467,7 +469,7 @@ unlink_free_block(Allctr_t *allctr, Block_t *block) } static void -update_last_aux_mbc(Allctr_t *allctr, Carrier_t *mbc) +update_last_aux_mbc(Allctr_t *allctr, Carrier_t *mbc, Uint32 flags) { GFAllctr_t *gfallctr = (GFAllctr_t *) allctr; diff --git a/erts/emulator/beam/erl_instrument.c b/erts/emulator/beam/erl_instrument.c index f3f3c22933..04ea004ef7 100644 --- a/erts/emulator/beam/erl_instrument.c +++ b/erts/emulator/beam/erl_instrument.c @@ -1186,6 +1186,8 @@ erts_instr_init(int stat, int map_stat) sys_memzero((void *) stats->n, sizeof(Stat_t)*(ERTS_ALC_N_MAX+1)); for (i = ERTS_ALC_A_MIN; i <= ERTS_ALC_A_MAX; i++) { + if (ERTS_IS_SBMBC_ALLOCATOR_NO__(i)) + continue; if (erts_allctrs_info[i].enabled) stats->ap[i] = &stats->a[i]; else @@ -1199,6 +1201,8 @@ erts_instr_init(int stat, int map_stat) erts_instr_memory_map = 1; erts_instr_stat = 1; for (i = ERTS_ALC_A_MIN; i <= ERTS_ALC_A_MAX; i++) { + if (ERTS_IS_SBMBC_ALLOCATOR_NO__(i)) + continue; erts_allctrs[i].alloc = map_stat_alloc; erts_allctrs[i].realloc = map_stat_realloc; erts_allctrs[i].free = map_stat_free; @@ -1209,6 +1213,8 @@ erts_instr_init(int stat, int map_stat) else { erts_instr_stat = 1; for (i = ERTS_ALC_A_MIN; i <= ERTS_ALC_A_MAX; i++) { + if (ERTS_IS_SBMBC_ALLOCATOR_NO__(i)) + continue; erts_allctrs[i].alloc = stat_alloc; erts_allctrs[i].realloc = stat_realloc; erts_allctrs[i].free = stat_free; diff --git a/erts/emulator/beam/erl_lock_check.c b/erts/emulator/beam/erl_lock_check.c index 9180508a49..587d82f2bb 100644 --- a/erts/emulator/beam/erl_lock_check.c +++ b/erts/emulator/beam/erl_lock_check.c @@ -153,6 +153,7 @@ static erts_lc_lock_order_t erts_lock_order[] = { { "instr", NULL }, { "fix_alloc", "index" }, { "alcu_allocator", "index" }, + { "sbmbc_alloc", "index" }, { "alcu_delayed_free", "index" }, { "mseg", NULL }, #if HALFWORD_HEAP diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c index 68421b4387..d9b1a8e89d 100644 --- a/erts/emulator/beam/erl_nif.c +++ b/erts/emulator/beam/erl_nif.c @@ -833,8 +833,11 @@ ERL_NIF_TERM enif_make_uint(ErlNifEnv* env, unsigned i) ERL_NIF_TERM enif_make_long(ErlNifEnv* env, long i) { + if (IS_SSMALL(i)) { + return make_small(i); + } #if SIZEOF_LONG == ERTS_SIZEOF_ETERM - return IS_SSMALL(i) ? make_small(i) : small_to_big(i, alloc_heap(env,2)); + return small_to_big(i, alloc_heap(env,2)); #elif SIZEOF_LONG == 8 ensure_heap(env,3); return erts_sint64_to_big(i, &env->hp); @@ -843,8 +846,11 @@ ERL_NIF_TERM enif_make_long(ErlNifEnv* env, long i) ERL_NIF_TERM enif_make_ulong(ErlNifEnv* env, unsigned long i) { + if (IS_USMALL(0,i)) { + return make_small(i); + } #if SIZEOF_LONG == ERTS_SIZEOF_ETERM - return IS_USMALL(0,i) ? make_small(i) : uint_to_big(i,alloc_heap(env,2)); + return uint_to_big(i,alloc_heap(env,2)); #elif SIZEOF_LONG == 8 ensure_heap(env,3); return erts_uint64_to_big(i, &env->hp); diff --git a/erts/emulator/beam/ops.tab b/erts/emulator/beam/ops.tab index 8a5763b4bb..304ce22ef2 100644 --- a/erts/emulator/beam/ops.tab +++ b/erts/emulator/beam/ops.tab @@ -1236,7 +1236,7 @@ i_bs_init_heap I I I d i_bs_init_heap_bin_heap I I I d -bs_init_bits Fail Sz Words Regs Flags Dst | binary_too_big_bits(Sz) => system_limit Fail +bs_init_bits Fail Sz=o Words Regs Flags Dst => system_limit Fail bs_init_bits Fail Sz=u Words=u==0 Regs Flags Dst => i_bs_init_bits Sz Regs Dst bs_init_bits Fail Sz=u Words Regs Flags Dst => i_bs_init_bits_heap Sz Words Regs Dst diff --git a/erts/emulator/drivers/common/inet_drv.c b/erts/emulator/drivers/common/inet_drv.c index 40c4a0df08..ebc4469a23 100644 --- a/erts/emulator/drivers/common/inet_drv.c +++ b/erts/emulator/drivers/common/inet_drv.c @@ -3709,6 +3709,8 @@ static int inet_ctl_fdopen(inet_descriptor* desc, int domain, int type, /* check that it is a socket and that the socket is bound */ if (IS_SOCKET_ERROR(sock_name(s, (struct sockaddr*) &name, &sz))) return ctl_error(sock_errno(), rbuf, rsize); + if (name.sa.sa_family != domain) + return ctl_error(EINVAL, rbuf, rsize); desc->s = s; if ((desc->event = sock_create_event(desc)) == INVALID_EVENT) return ctl_error(sock_errno(), rbuf, rsize); @@ -9739,7 +9741,7 @@ static int packet_inet_ctl(ErlDrvData e, unsigned int cmd, char* buf, int len, if (desc->active || (len != 8)) return ctl_error(EINVAL, rbuf, rsize); timeout = get_int32(buf); - /* The 2nd arg, Length(4), is ignored for both UDP ans SCTP protocols, + /* The 2nd arg, Length(4), is ignored for both UDP and SCTP protocols, since they are msg-oriented. */ if (enq_async(desc, tbuf, PACKET_REQ_RECV) < 0) diff --git a/erts/emulator/hipe/hipe_mode_switch.c b/erts/emulator/hipe/hipe_mode_switch.c index 16f8fb1347..e3e8367b62 100644 --- a/erts/emulator/hipe/hipe_mode_switch.c +++ b/erts/emulator/hipe/hipe_mode_switch.c @@ -346,7 +346,12 @@ Process *hipe_mode_switch(Process *p, unsigned cmd, Eterm reg[]) p->arity = callee_arity; } - /* If process is in P_WAITING state, we schedule the next process */ + /* Schedule next process if current process was hibernated or is waiting + for messages */ + if (p->flags & F_HIBERNATE_SCHED) { + p->flags &= ~F_HIBERNATE_SCHED; + goto do_schedule; + } if (p->status == P_WAITING) { goto do_schedule; } diff --git a/erts/emulator/test/alloc_SUITE_data/allocator_test.h b/erts/emulator/test/alloc_SUITE_data/allocator_test.h index b869a4079c..8b34375980 100644 --- a/erts/emulator/test/alloc_SUITE_data/allocator_test.h +++ b/erts/emulator/test/alloc_SUITE_data/allocator_test.h @@ -82,15 +82,17 @@ typedef void* erts_cond; #define NO_OF_BKTS ((Ulong) ALC_TEST0(0x102)) #define FIND_BKT(A, I) ((int) ALC_TEST2(0x103, (A), (I))) -/* From erl_bestfit_alloc.c */ -#define IS_AOBF(A) ((Ulong) ALC_TEST1(0x200, (A))) -#define RBT_ROOT(A) ((RBT_t *) ALC_TEST1(0x201, (A))) -#define RBT_PARENT(T) ((RBT_t *) ALC_TEST1(0x202, (T))) -#define RBT_LEFT(T) ((RBT_t *) ALC_TEST1(0x203, (T))) -#define RBT_RIGHT(T) ((RBT_t *) ALC_TEST1(0x204, (T))) -#define RBT_NEXT(T) ((RBTL_t *) ALC_TEST1(0x205, (T))) -#define RBT_IS_BLACK(T) ((Ulong) ALC_TEST1(0x206, (T))) -#define RBT_IS_TREE(T) ((Ulong) ALC_TEST1(0x207, (T))) +/* From erl_bestfit_alloc.c and erl_ao_firstfit_alloc.c */ +#define IS_AOBF(A) ((Ulong) ALC_TEST1(RBT_OP(0), (A))) +#define RBT_ROOT(A) ((RBT_t *) ALC_TEST1(RBT_OP(1), (A))) +#define RBT_PARENT(T) ((RBT_t *) ALC_TEST1(RBT_OP(2), (T))) +#define RBT_LEFT(T) ((RBT_t *) ALC_TEST1(RBT_OP(3), (T))) +#define RBT_RIGHT(T) ((RBT_t *) ALC_TEST1(RBT_OP(4), (T))) +#define RBT_NEXT(T) ((RBTL_t *) ALC_TEST1(RBT_OP(5), (T))) +#define RBT_IS_BLACK(T) ((Ulong) ALC_TEST1(RBT_OP(6), (T))) +#define RBT_IS_TREE(T) ((Ulong) ALC_TEST1(RBT_OP(7), (T))) +#define IS_AOFF(A) ((Ulong) ALC_TEST1(RBT_OP(8), (A))) +#define RBT_MAX_SZ(T) ((Ulong) ALC_TEST1(RBT_OP(9), (T))) /* From erl_mseg.c */ #define HAVE_MSEG() ((int) ALC_TEST0(0x400)) diff --git a/erts/emulator/test/alloc_SUITE_data/coalesce.c b/erts/emulator/test/alloc_SUITE_data/coalesce.c index c84da97d35..6f35d3279b 100644 --- a/erts/emulator/test/alloc_SUITE_data/coalesce.c +++ b/erts/emulator/test/alloc_SUITE_data/coalesce.c @@ -267,7 +267,7 @@ void testcase_run(TestCaseState_t *tcs) { char *argv_org[] = {"-tmmbcs1024", "-tsbct2048", "-trmbcmt100", "-tas", NULL, NULL}; - char *alg[] = {"af", "gf", "bf", "aobf", NULL}; + char *alg[] = {"af", "gf", "bf", "aobf", "aoff", NULL}; int i; for (i = 0; alg[i]; i++) { diff --git a/erts/emulator/test/alloc_SUITE_data/rbtree.c b/erts/emulator/test/alloc_SUITE_data/rbtree.c index c97e0aac1a..4e7f821baf 100644 --- a/erts/emulator/test/alloc_SUITE_data/rbtree.c +++ b/erts/emulator/test/alloc_SUITE_data/rbtree.c @@ -34,6 +34,14 @@ typedef struct { #define PRINT_TREE #endif +/* Ugly hack to steer the test code towards the right allocator */ +#define RBT_OP(CMD) (current_rbt_type_op_base + (CMD)) +static enum { + BESTFIT_OP_BASE = 0x200, + AO_FIRSTFIT_OP_BASE = 0x500 +}current_rbt_type_op_base; + + #ifdef PRINT_TREE #define INDENT_STEP 5 @@ -65,12 +73,11 @@ print_tree_aux(TestCaseState_t *tcs, RBT_t *x, int indent) static void -print_tree(TestCaseState_t *tcs, RBT_t *root, int aobf) +print_tree(TestCaseState_t *tcs, RBT_t *root) { - char *type = aobf ? "Size-Adress" : "Size"; - testcase_printf(tcs, " --- %s tree begin ---\r\n", type); + testcase_printf(tcs, " --- Tree begin ---\r\n"); print_tree_aux(tcs, root, 0); - testcase_printf(tcs, " --- %s tree end ---\r\n", type); + testcase_printf(tcs, " --- Tree end ---\r\n"); } #endif @@ -78,7 +85,8 @@ print_tree(TestCaseState_t *tcs, RBT_t *root, int aobf) static RBT_t * check_tree(TestCaseState_t *tcs, Allctr_t *alc, Ulong size) { - int i, max_i, address_order; + enum { BF, AOBF, AOFF }type; + int i, max_i; char stk[128]; RBT_t *root, *x, *y, *res; Ulong x_sz, y_sz, is_x_black; @@ -86,11 +94,14 @@ check_tree(TestCaseState_t *tcs, Allctr_t *alc, Ulong size) res = NULL; - address_order = IS_AOBF(alc); + if (IS_AOBF(alc)) type = AOBF; + else if (IS_AOFF(alc)) type = AOFF; + else type = BF; + root = RBT_ROOT(alc); #ifdef PRINT_TREE - print_tree(tcs, root, address_order); + print_tree(tcs, root); #endif max_i = i = -1; @@ -165,12 +176,18 @@ check_tree(TestCaseState_t *tcs, Allctr_t *alc, Ulong size) if (y) { y_sz = BLK_SZ(y); ASSERT(tcs, RBT_PARENT(y) == x); - if (address_order) { + switch (type) { + case AOBF: ASSERT(tcs, y_sz < x_sz || (y_sz == x_sz && y < x)); - } - else { + break; + case BF: ASSERT(tcs, RBT_IS_TREE(y)); ASSERT(tcs, y_sz < x_sz); + break; + case AOFF: + ASSERT(tcs, y < x); + ASSERT(tcs, RBT_MAX_SZ(y) <= RBT_MAX_SZ(x)); + break; } } @@ -178,16 +195,22 @@ check_tree(TestCaseState_t *tcs, Allctr_t *alc, Ulong size) if (y) { y_sz = BLK_SZ(y); ASSERT(tcs, RBT_PARENT(y) == x); - if (address_order) { + switch (type) { + case AOBF: ASSERT(tcs, y_sz > x_sz || (y_sz == x_sz && y > x)); - } - else { + break; + case BF: ASSERT(tcs, RBT_IS_TREE(y)); ASSERT(tcs, y_sz > x_sz); + break; + case AOFF: + ASSERT(tcs, y > x); + ASSERT(tcs, RBT_MAX_SZ(y) <= RBT_MAX_SZ(x)); + break; } } - if (!address_order) { + if (type == BF) { Ulong l_sz; RBTL_t *l = RBT_NEXT(x); for (l = RBT_NEXT(x); l; l = RBT_NEXT(l)) { @@ -202,13 +225,20 @@ check_tree(TestCaseState_t *tcs, Allctr_t *alc, Ulong size) res = x; else { y_sz = BLK_SZ(res); - if (address_order) { + switch (type) { + case AOBF: if (x_sz < y_sz || (x_sz == y_sz && x < res)) res = x; - } - else { - if (!res || x_sz < y_sz) + break; + case BF: + if (x_sz < y_sz) res = x; + break; + case AOFF: + if (x < res) { + res = x; + } + break; } } } @@ -257,7 +287,7 @@ static void test_it(TestCaseState_t *tcs) { int i; - Allctr_t a = ((rbtree_test_data *) tcs->extra)->allocator; + Allctr_t* a = ((rbtree_test_data *) tcs->extra)->allocator; void **blk = ((rbtree_test_data *) tcs->extra)->blk; void **fence = ((rbtree_test_data *) tcs->extra)->fence; Ulong min_blk_sz; @@ -338,6 +368,7 @@ testcase_run(TestCaseState_t *tcs) { char *argv1[] = {"-tasbf", NULL}; char *argv2[] = {"-tasaobf", NULL}; + char *argv3[] = {"-tasaoff", NULL}; Allctr_t *a; rbtree_test_data *td; @@ -355,6 +386,7 @@ testcase_run(TestCaseState_t *tcs) testcase_printf(tcs, "Starting test of best fit...\n"); + current_rbt_type_op_base = BESTFIT_OP_BASE; td->allocator = a = START_ALC("rbtree_bf_", 0, argv1); ASSERT(tcs, a); @@ -371,6 +403,7 @@ testcase_run(TestCaseState_t *tcs) testcase_printf(tcs, "Starting test of address order best fit...\n"); + current_rbt_type_op_base = BESTFIT_OP_BASE; td->allocator = a = START_ALC("rbtree_aobf_", 0, argv2); ASSERT(tcs, a); @@ -383,4 +416,19 @@ testcase_run(TestCaseState_t *tcs) testcase_printf(tcs, "Address order best fit test succeeded!\n"); + /* Address order first fit... */ + + testcase_printf(tcs, "Starting test of address order first fit...\n"); + + current_rbt_type_op_base = AO_FIRSTFIT_OP_BASE; + td->allocator = a = START_ALC("rbtree_aoff_", 0, argv3); + + ASSERT(tcs, a); + + test_it(tcs); + + STOP_ALC(a); + td->allocator = NULL; + + testcase_printf(tcs, "Address order first fit test succeeded!\n"); } diff --git a/erts/emulator/test/bs_construct_SUITE.erl b/erts/emulator/test/bs_construct_SUITE.erl index 1959803385..7fdf36711b 100644 --- a/erts/emulator/test/bs_construct_SUITE.erl +++ b/erts/emulator/test/bs_construct_SUITE.erl @@ -553,6 +553,11 @@ huge_float_check({'EXIT',{badarg,_}}) -> ok. huge_binary(Config) when is_list(Config) -> ?line 16777216 = size(<<0:(id(1 bsl 26)),(-1):(id(1 bsl 26))>>), + ?line garbage_collect(), + ?line id(<<0:((1 bsl 32)-1)>>), + ?line garbage_collect(), + ?line id(<<0:(id((1 bsl 32)-1))>>), + ?line garbage_collect(), ok. system_limit(Config) when is_list(Config) -> @@ -565,6 +570,10 @@ system_limit(Config) when is_list(Config) -> ?line {'EXIT',{system_limit,_}} = (catch <<(id(<<>>))/binary,0:(id(1 bsl 100))>>), + %% Would fail to load. + ?line {'EXIT',{system_limit,_}} = (catch <<0:(1 bsl 67)>>), + ?line {'EXIT',{system_limit,_}} = (catch <<0:((1 bsl 64)+1)>>), + case WordSize of 4 -> system_limit_32(); @@ -581,6 +590,14 @@ system_limit_32() -> ?line {'EXIT',{system_limit,_}} = (catch <<0:(id(8)),42:536870912/unit:8>>), ?line {'EXIT',{system_limit,_}} = (catch <<0:(id(8)),42:(id(536870912))/unit:8>>), + + %% The size would be silently truncated, resulting in a crash. + ?line {'EXIT',{system_limit,_}} = (catch <<0:(1 bsl 35)>>), + ?line {'EXIT',{system_limit,_}} = (catch <<0:((1 bsl 32)+1)>>), + + %% Would fail to load. + ?line {'EXIT',{system_limit,_}} = (catch <<0:(1 bsl 43)>>), + ?line {'EXIT',{system_limit,_}} = (catch <<0:((1 bsl 40)+1)>>), ok. badarg(Config) when is_list(Config) -> diff --git a/erts/emulator/test/fun_SUITE.erl b/erts/emulator/test/fun_SUITE.erl index 7795efe57e..559e540016 100644 --- a/erts/emulator/test/fun_SUITE.erl +++ b/erts/emulator/test/fun_SUITE.erl @@ -647,17 +647,11 @@ refc_dist_1() -> %% Fun is passed in an exit signal. Wait until it is gone. ?line wait_until(fun () -> 4 =/= fun_refc(F2) end), ?line 3 = fun_refc(F2), - erts_debug:set_internal_state(available_internal_state, true), - ?line F_refc = case erts_debug:get_internal_state(force_heap_frags) of - false -> 3; - true -> 2 % GC after bif already decreased it - end, - ?line F_refc = fun_refc(F), - erts_debug:set_internal_state(available_internal_state, false), + ?line true = erlang:garbage_collect(), + ?line 2 = fun_refc(F), refc_dist_send(Node, F). refc_dist_send(Node, F) -> - ?line true = erlang:garbage_collect(), ?line Pid = spawn_link(Node, fun() -> receive {To,Fun} when is_function(Fun) -> diff --git a/erts/emulator/test/hibernate_SUITE.erl b/erts/emulator/test/hibernate_SUITE.erl index 203fa6b48e..82a0aad189 100644 --- a/erts/emulator/test/hibernate_SUITE.erl +++ b/erts/emulator/test/hibernate_SUITE.erl @@ -25,16 +25,16 @@ init_per_group/2,end_per_group/2, init_per_testcase/2,end_per_testcase/2, basic/1,dynamic_call/1,min_heap_size/1,bad_args/1, - messages_in_queue/1,undefined_mfa/1, no_heap/1]). + messages_in_queue/1,undefined_mfa/1,no_heap/1,wake_up_and_bif_trap/1]). %% Used by test cases. --export([basic_hibernator/1,dynamic_call_hibernator/2,messages_in_queue_restart/2, no_heap_loop/0]). +-export([basic_hibernator/1,dynamic_call_hibernator/2,messages_in_queue_restart/2, no_heap_loop/0,characters_to_list_trap/1]). suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [basic, dynamic_call, min_heap_size, bad_args, messages_in_queue, - undefined_mfa, no_heap]. + undefined_mfa, no_heap, wake_up_and_bif_trap]. groups() -> []. @@ -384,6 +384,31 @@ clean_dict() -> lists:foreach(fun ({Key, _}) -> erase(Key) end, Dict). %% +%% Wake up and then immediatly bif trap with a lengthy computation. +%% + +wake_up_and_bif_trap(doc) -> []; +wake_up_and_bif_trap(suite) -> []; +wake_up_and_bif_trap(Config) when is_list(Config) -> + ?line Self = self(), + ?line Pid = spawn_link(fun() -> erlang:hibernate(?MODULE, characters_to_list_trap, [Self]) end), + ?line Pid ! wakeup, + ?line receive + {ok, Pid0} when Pid0 =:= Pid -> ok + after 5000 -> + ?line ?t:fail(process_blocked) + end, + ?line unlink(Pid), + ?line exit(Pid, bye). + +%% Lengthy computation that traps (in characters_to_list_trap_3). +characters_to_list_trap(Parent) -> + Bin0 = <<"abcdefghijklmnopqrstuvwxz0123456789">>, + Bin = binary:copy(Bin0, 1500), + unicode:characters_to_list(Bin), + Parent ! {ok, self()}. + +%% %% Misc %% diff --git a/erts/emulator/test/match_spec_SUITE.erl b/erts/emulator/test/match_spec_SUITE.erl index 2b21fa58f4..461773114e 100644 --- a/erts/emulator/test/match_spec_SUITE.erl +++ b/erts/emulator/test/match_spec_SUITE.erl @@ -27,8 +27,9 @@ destructive_in_test_bif/1, guard_exceptions/1, unary_plus/1, unary_minus/1, moving_labels/1]). -export([fpe/1]). +-export([otp_9422/1]). --export([runner/2]). +-export([runner/2, loop_runner/3]). -export([f1/1, f2/2, f3/2, fn/1, fn/2, fn/3]). -export([do_boxed_and_small/0]). @@ -57,7 +58,8 @@ all() -> trace_control_word, silent, silent_no_ms, ms_trace2, ms_trace3, boxed_and_small, destructive_in_test_bif, guard_exceptions, unary_plus, unary_minus, fpe, - moving_labels]; + moving_labels, + otp_9422]; true -> [not_run] end. @@ -208,6 +210,43 @@ test_3(Config) when is_list(Config) -> ?line collect(P1, [{trace, P1, call, {?MODULE, f2, [a, b]}, [true]}]), ?line ok. +otp_9422(doc) -> []; +otp_9422(Config) when is_list(Config) -> + Laps = 1000, + ?line Fun1 = fun() -> otp_9422_tracee() end, + ?line P1 = spawn_link(?MODULE, loop_runner, [self(), Fun1, Laps]), + io:format("spawned ~p as tracee\n", [P1]), + + ?line erlang:trace(P1, true, [call, silent]), + + ?line Fun2 = fun() -> otp_9422_trace_changer() end, + ?line P2 = spawn_link(?MODULE, loop_runner, [self(), Fun2, Laps]), + io:format("spawned ~p as trace_changer\n", [P2]), + + start_collect(P1), + start_collect(P2), + + %%receive after 10*1000 -> ok end, + + stop_collect(P1), + stop_collect(P2), + ok. + +otp_9422_tracee() -> + ?MODULE:f1(a), + ?MODULE:f1(b), + ?MODULE:f1(c). + +otp_9422_trace_changer() -> + Pat1 = [{[a], [], [{enable_trace, arity}]}], + ?line erlang:trace_pattern({?MODULE, f1, 1}, Pat1), + Pat2 = [{[b], [], [{disable_trace, arity}]}], + ?line erlang:trace_pattern({?MODULE, f1, 1}, Pat2). + + + + + bad_match_spec_bin(Config) when is_list(Config) -> {'EXIT',{badarg,_}} = (catch ets:match_spec_run([1], <<>>)), B0 = <<1,2>>, @@ -932,6 +971,24 @@ runner(Collector, Fun) -> Collector ! {gone, self()} end. +loop_runner(Collector, Fun, Laps) -> + receive + {go, Collector} -> + go + end, + loop_runner_cont(Collector, Fun, 0, Laps). + +loop_runner_cont(_Collector, _Fun, Laps, Laps) -> + receive + {done, Collector} -> + io:format("loop_runner ~p exit after ~p laps\n", [self(), Laps]), + Collector ! {gone, self()} + end; +loop_runner_cont(Collector, Fun, N, Laps) -> + Fun(), + loop_runner_cont(Collector, Fun, N+1, Laps). + + f1(X) -> {X}. diff --git a/erts/emulator/test/nif_SUITE.erl b/erts/emulator/test/nif_SUITE.erl index 91d695d979..2867e8e2e4 100644 --- a/erts/emulator/test/nif_SUITE.erl +++ b/erts/emulator/test/nif_SUITE.erl @@ -257,10 +257,48 @@ types(Config) when is_list(Config) -> end, [{},{ok},{{}},{[],{}},{1,2,3,4,5}]), Stuff = [[],{},0,0.0,(1 bsl 100),(fun()-> ok end),make_ref(),self()], - [eq_cmp(A,clone(B)) || A<-Stuff, B<-Stuff], + [eq_cmp(A,clone(B)) || A<-Stuff, B<-Stuff], + + {IntSz, LongSz} = type_sizes(), + UintMax = (1 bsl (IntSz*8)) - 1, + IntMax = UintMax bsr 1, + IntMin = -(IntMax+1), + UlongMax = (1 bsl (LongSz*8)) - 1, + LongMax = UlongMax bsr 1, + LongMin = -(LongMax+1), + Uint64Max = (1 bsl 64) - 1, + Int64Max = Uint64Max bsr 1, + Int64Min = -(Int64Max+1), + Limits = [{IntMin,IntMax},{0,UintMax},{LongMin,LongMax},{0,UlongMax},{Int64Min,Int64Max},{0,Uint64Max}], + io:format("Limits = ~p\n", [Limits]), + lists:foreach(fun(I) -> + R1 = echo_int(I), + %%io:format("echo_int(~p) -> ~p\n", [I, R1]), + R2 = my_echo_int(I, Limits), + ?line R1 = R2, + ?line true = (R1 =:= R2), + ?line true = (R1 == R2) + end, int_list()), + ?line verify_tmpmem(TmpMem), ok. +int_list() -> + Start = 1 bsl 200, + int_list([Start], -Start). +int_list([N | _]=List, End) when N<End -> + List; +int_list([N | _]=List, End) -> + int_list([N - (1 + (abs(N) div 3)) | List], End). + +my_echo_int(I, Limits) -> + lists:map(fun({Min,Max}) -> + if I < Min -> false; + I > Max -> false; + true -> I + end + end, Limits). + clone(X) -> binary_to_term(term_to_binary(X)). @@ -1170,9 +1208,10 @@ tmpmem() -> MemInfo -> MSBCS = lists:foldl( fun ({instance, _, L}, Acc) -> + {value,{_,SBMBCS}} = lists:keysearch(sbmbcs, 1, L), {value,{_,MBCS}} = lists:keysearch(mbcs, 1, L), {value,{_,SBCS}} = lists:keysearch(sbcs, 1, L), - [MBCS,SBCS | Acc] + [SBMBCS,MBCS,SBCS | Acc] end, [], MemInfo), @@ -1269,6 +1308,8 @@ send_blob_thread(_,_,_) -> ?nif_stub. join_send_thread(_) -> ?nif_stub. copy_blob(_) -> ?nif_stub. send_term(_,_) -> ?nif_stub. +echo_int(_) -> ?nif_stub. +type_sizes() -> ?nif_stub. nif_stub_error(Line) -> exit({nif_not_loaded,module,?MODULE,line,Line}). diff --git a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c index 0bb93daa33..00a1365bc3 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c +++ b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c @@ -28,6 +28,7 @@ static int static_cntA; /* zero by default */ static int static_cntB = NIF_SUITE_LIB_VER * 100; +static ERL_NIF_TERM atom_false; static ERL_NIF_TERM atom_self; static ERL_NIF_TERM atom_ok; static ERL_NIF_TERM atom_join; @@ -103,7 +104,7 @@ static int load(ErlNifEnv* env, void** priv_data, ERL_NIF_TERM load_info) msgenv_resource_type = enif_open_resource_type(env,NULL,"nif_SUITE.msgenv", msgenv_dtor, ERL_NIF_RT_CREATE, NULL); - + atom_false = enif_make_atom(env,"false"); atom_self = enif_make_atom(env,"self"); atom_ok = enif_make_atom(env,"ok"); atom_join = enif_make_atom(env,"join"); @@ -481,6 +482,45 @@ error: return enif_make_atom(env,"error"); } +static ERL_NIF_TERM echo_int(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + int sint; + unsigned uint; + long slong; + unsigned long ulong; + ErlNifSInt64 sint64; + ErlNifUInt64 uint64; + ERL_NIF_TERM sint_term = atom_false, uint_term = atom_false; + ERL_NIF_TERM slong_term = atom_false, ulong_term = atom_false; + ERL_NIF_TERM sint64_term = atom_false, uint64_term = atom_false; + + if (enif_get_int(env, argv[0], &sint)) { + sint_term = enif_make_int(env, sint); + } + if (enif_get_uint(env, argv[0], &uint)) { + uint_term = enif_make_uint(env, uint); + } + if (enif_get_long(env, argv[0], &slong)) { + slong_term = enif_make_long(env, slong); + } + if (enif_get_ulong(env, argv[0], &ulong)) { + ulong_term = enif_make_ulong(env, ulong); + } + if (enif_get_int64(env, argv[0], &sint64)) { + sint64_term = enif_make_int64(env, sint64); + } + if (enif_get_uint64(env, argv[0], &uint64)) { + uint64_term = enif_make_uint64(env, uint64); + } + return enif_make_list6(env, sint_term, uint_term, slong_term, ulong_term, sint64_term, uint64_term); +} + +static ERL_NIF_TERM type_sizes(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) +{ + return enif_make_tuple2(env, enif_make_int(env, sizeof(int)), + enif_make_int(env, sizeof(long))); +} + static ERL_NIF_TERM tuple_2_list(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) { int arity = -1; @@ -1417,7 +1457,9 @@ static ErlNifFunc nif_funcs[] = {"send_blob_thread", 3, send_blob_thread}, {"join_send_thread", 1, join_send_thread}, {"copy_blob", 1, copy_blob}, - {"send_term", 2, send_term} + {"send_term", 2, send_term}, + {"echo_int", 1, echo_int}, + {"type_sizes", 0, type_sizes} }; ERL_NIF_INIT(nif_SUITE,nif_funcs,load,reload,upgrade,unload) diff --git a/erts/emulator/test/send_term_SUITE.erl b/erts/emulator/test/send_term_SUITE.erl index 6615873392..ba0ba804ca 100644 --- a/erts/emulator/test/send_term_SUITE.erl +++ b/erts/emulator/test/send_term_SUITE.erl @@ -175,6 +175,10 @@ chk_temp_alloc() -> %% Verify that we havn't got anything allocated by temp_alloc lists:foreach( fun ({instance, _, TI}) -> + ?line {value, {sbmbcs, SBMBCInfo}} + = lists:keysearch(sbmbcs, 1, TI), + ?line {value, {blocks, 0, _, _}} + = lists:keysearch(blocks, 1, SBMBCInfo), ?line {value, {mbcs, MBCInfo}} = lists:keysearch(mbcs, 1, TI), ?line {value, {blocks, 0, _, _}} diff --git a/erts/emulator/utils/beam_makeops b/erts/emulator/utils/beam_makeops index e7c57142c0..354439b5e3 100755 --- a/erts/emulator/utils/beam_makeops +++ b/erts/emulator/utils/beam_makeops @@ -67,6 +67,10 @@ my $max_gen_operands = 8; # Must be even. The beam_load.c file must be updated, too. my $max_spec_operands = 6; +# The maximum number of primitive genop_types. + +my $max_genop_types = 16; + my %gen_opnum; my %num_specific; my %gen_to_spec; @@ -106,7 +110,7 @@ my @pred_table; # Operand types for generic instructions. my $compiler_types = "uiaxyfhz"; -my $loader_types = "nprvlq"; +my $loader_types = "nprvlqo"; my $genop_types = $compiler_types . $loader_types; # @@ -142,34 +146,61 @@ my %arg_size = ('r' => 0, # x(0) - x register zero my %type_bit; my @tag_type; +sub define_type_bit { + my($tag,$val) = @_; + defined $type_bit{$tag} and + sanity("the tag '$tag' has already been defined with the value ", + $type_bit{$tag}); + $type_bit{$tag} = $val; +} + { my($bit) = 1; my(%bit); foreach (split('', $genop_types)) { push(@tag_type, $_); - $type_bit{$_} = $bit; + define_type_bit($_, $bit); $bit{$_} = $bit; $bit *= 2; } # Composed types. - $type_bit{'d'} = $type_bit{'x'} | $type_bit{'y'} | $type_bit{'r'}; - $type_bit{'c'} = $type_bit{'i'} | $type_bit{'a'} | $type_bit{'n'} | $type_bit{'q'}; - $type_bit{'s'} = $type_bit{'d'} | $type_bit{'i'} | $type_bit{'a'} | $type_bit{'n'}; - $type_bit{'j'} = $type_bit{'f'} | $type_bit{'p'}; + define_type_bit('d', $type_bit{'x'} | $type_bit{'y'} | $type_bit{'r'}); + define_type_bit('c', $type_bit{'i'} | $type_bit{'a'} | + $type_bit{'n'} | $type_bit{'q'}); + define_type_bit('s', $type_bit{'d'} | $type_bit{'i'} | + $type_bit{'a'} | $type_bit{'n'}); + define_type_bit('j', $type_bit{'f'} | $type_bit{'p'}); # Aliases (for matching purposes). - $type_bit{'I'} = $type_bit{'u'}; - $type_bit{'t'} = $type_bit{'u'}; - $type_bit{'A'} = $type_bit{'u'}; - $type_bit{'L'} = $type_bit{'u'}; - $type_bit{'b'} = $type_bit{'u'}; - $type_bit{'N'} = $type_bit{'u'}; - $type_bit{'U'} = $type_bit{'u'}; - $type_bit{'e'} = $type_bit{'u'}; - $type_bit{'P'} = $type_bit{'u'}; - $type_bit{'Q'} = $type_bit{'u'}; + define_type_bit('I', $type_bit{'u'}); + define_type_bit('t', $type_bit{'u'}); + define_type_bit('A', $type_bit{'u'}); + define_type_bit('L', $type_bit{'u'}); + define_type_bit('b', $type_bit{'u'}); + define_type_bit('N', $type_bit{'u'}); + define_type_bit('U', $type_bit{'u'}); + define_type_bit('e', $type_bit{'u'}); + define_type_bit('P', $type_bit{'u'}); + define_type_bit('Q', $type_bit{'u'}); +} + +# +# Sanity checks. +# + +{ + if (@tag_type > $max_genop_types) { + sanity("\$max_genop_types is $max_genop_types, ", + "but there are ", scalar(@tag_type), + " primitive tags defined\n"); + } + + foreach my $tag (@tag_type) { + sanity("tag '$tag': primitive tags must be named with lowercase letters") + unless $tag =~ /^[a-z]$/; + } } # @@ -436,12 +467,12 @@ sub emulator_output { # my(@bits) = (0) x ($max_spec_operands/2); - my($shift) = 16; my($i); for ($i = 0; $i < $max_spec_operands && defined $args[$i]; $i++) { my $t = $args[$i]; if (defined $type_bit{$t}) { - $bits[int($i/2)] |= $type_bit{$t} << (16*($i%2)); + my $shift = $max_genop_types * ($i % 2); + $bits[int($i/2)] |= $type_bit{$t} << $shift; } } @@ -753,6 +784,10 @@ sub error { die $where, @message, "\n"; } +sub sanity { + die "internal error: ", @_, "\n"; +} + sub comment { my($lang, @comments) = @_; my($prefix); diff --git a/erts/epmd/src/epmd_int.h b/erts/epmd/src/epmd_int.h index 2a0de4df9c..a2d7559f9d 100644 --- a/erts/epmd/src/epmd_int.h +++ b/erts/epmd/src/epmd_int.h @@ -240,6 +240,14 @@ #define put_int16(i, s) {((unsigned char*)(s))[0] = ((i) >> 8) & 0xff; \ ((unsigned char*)(s))[1] = (i) & 0xff;} +#if defined(__GNUC__) +# define EPMD_INLINE __inline__ +#elif defined(__WIN32__) +# define EPMD_INLINE __inline +#else +# define EPMD_INLINE +#endif + /* ************************************************************************ */ /* Stuctures used by server */ @@ -295,6 +303,7 @@ typedef struct { unsigned delay_write; int max_conn; int active_conn; + int select_fd_top; char *progname; Connection *conn; Nodes nodes; diff --git a/erts/epmd/src/epmd_srv.c b/erts/epmd/src/epmd_srv.c index 4d9b454f97..5debae26b6 100644 --- a/erts/epmd/src/epmd_srv.c +++ b/erts/epmd/src/epmd_srv.c @@ -80,6 +80,13 @@ static int reply(EpmdVars*,int,char *,int); static void dbg_print_buf(EpmdVars*,char *,int); static void print_names(EpmdVars*); +static EPMD_INLINE void select_fd_set(EpmdVars* g, int fd) +{ + FD_SET(fd, &g->orig_read_mask); + if (fd >= g->select_fd_top) { + g->select_fd_top = fd + 1; + } +} void run(EpmdVars *g) { @@ -171,6 +178,7 @@ void run(EpmdVars *g) g->max_conn -= num_sockets; FD_ZERO(&g->orig_read_mask); + g->select_fd_top = 0; for (i = 0; i < num_sockets; i++) { @@ -232,14 +240,14 @@ void run(EpmdVars *g) dbg_perror(g,"failed to listen on socket"); epmd_cleanup_exit(g,1); } - FD_SET(listensock[i],&g->orig_read_mask); + select_fd_set(g, listensock[i]); } dbg_tty_printf(g,2,"entering the main select() loop"); select_again: while(1) - { + { fd_set read_mask = g->orig_read_mask; struct timeval timeout; int ret; @@ -251,7 +259,8 @@ void run(EpmdVars *g) timeout.tv_sec = (g->packet_timeout < IDLE_TIMEOUT) ? 1 : IDLE_TIMEOUT; timeout.tv_usec = 0; - if ((ret = select(g->max_conn,&read_mask,(fd_set *)0,(fd_set *)0,&timeout)) < 0) { + if ((ret = select(g->select_fd_top, + &read_mask, (fd_set *)0,(fd_set *)0,&timeout)) < 0) { dbg_perror(g,"error in select "); switch (errno) { case EAGAIN: @@ -821,7 +830,7 @@ static int conn_open(EpmdVars *g,int fd) s = &g->conn[i]; /* From now on we want to know if there are data to be read */ - FD_SET(fd, &g->orig_read_mask); + select_fd_set(g, fd); s->fd = fd; s->open = EPMD_TRUE; @@ -886,6 +895,7 @@ int epmd_conn_close(EpmdVars *g,Connection *s) dbg_tty_printf(g,2,"closing connection on file descriptor %d",s->fd); FD_CLR(s->fd,&g->orig_read_mask); + /* we don't bother lowering g->select_fd_top */ close(s->fd); /* Sometimes already closed but close anyway */ s->open = EPMD_FALSE; if (s->buf != NULL) { /* Should never be NULL but test anyway */ @@ -1115,7 +1125,7 @@ static Node *node_reg2(EpmdVars *g, node->extralen = extralen; memcpy(node->extra,extra,extralen); strcpy(node->symname,name); - FD_SET(fd,&g->orig_read_mask); + select_fd_set(g, fd); if (highvsn == 0) { dbg_tty_printf(g,1,"registering '%s:%d', port %d", diff --git a/erts/etc/common/erlexec.c b/erts/etc/common/erlexec.c index 7e04724f5a..2bd576d8e8 100644 --- a/erts/etc/common/erlexec.c +++ b/erts/etc/common/erlexec.c @@ -64,6 +64,7 @@ static const char plusM_au_allocs[]= { 'u', /* all alloc_util allocators */ 'B', /* binary_alloc */ + 'C', /* sbmbc_alloc */ 'D', /* std_alloc */ 'E', /* ets_alloc */ 'H', /* eheap_alloc */ @@ -93,6 +94,8 @@ static char *plusM_au_alloc_switches[] = { "rsbcst", "sbct", "smbcs", + "sbmbcs", + "sbmbct", NULL }; diff --git a/erts/lib_src/common/erl_misc_utils.c b/erts/lib_src/common/erl_misc_utils.c index 5dbf98c7d1..35b148990a 100644 --- a/erts/lib_src/common/erl_misc_utils.c +++ b/erts/lib_src/common/erl_misc_utils.c @@ -1511,7 +1511,7 @@ const char* parse_topology_spec_group(erts_cpu_info_t *cpuinfo, const char* xml, } } - if (cacheLevel == 0) { + if (parentCacheLevel == 0) { *core_p = 0; *processor_p = (*processor_p)++; } else { diff --git a/erts/lib_src/common/erl_printf_format.c b/erts/lib_src/common/erl_printf_format.c index fba3fd723c..473791dce4 100644 --- a/erts/lib_src/common/erl_printf_format.c +++ b/erts/lib_src/common/erl_printf_format.c @@ -388,7 +388,7 @@ static int fmt_double(fmtfn_t fn,void*arg,double val, max_size++; if (precision) max_size += precision; - else if (fmt && FMTF_alt) + else if (fmt & FMTF_alt) max_size++; break; case FMTC_E: @@ -402,7 +402,7 @@ static int fmt_double(fmtfn_t fn,void*arg,double val, max_size += 4; if (precision) max_size += precision; - else if (fmt && FMTF_alt) + else if (fmt & FMTF_alt) max_size++; aexp = exp >= 0 ? exp : -exp; if (aexp < 100) diff --git a/lib/edoc/Makefile b/lib/edoc/Makefile index e512e390e3..1add669398 100644 --- a/lib/edoc/Makefile +++ b/lib/edoc/Makefile @@ -13,8 +13,6 @@ # Portions created by Ericsson are Copyright 1999, Ericsson Utvecklings # AB. All Rights Reserved.'' # -# $Id$ -# include $(ERL_TOP)/make/target.mk include $(ERL_TOP)/make/$(TARGET)/otp.mk diff --git a/lib/edoc/doc/Makefile b/lib/edoc/doc/Makefile index a0f6484382..c5f68b25d0 100644 --- a/lib/edoc/doc/Makefile +++ b/lib/edoc/doc/Makefile @@ -13,8 +13,6 @@ # Portions created by Ericsson are Copyright 1999, Ericsson Utvecklings # AB. All Rights Reserved.'' # -# $Id: Makefile,v 1.1.1.1 2004/10/04 13:53:33 richardc Exp $ -# include $(ERL_TOP)/make/target.mk include $(ERL_TOP)/make/$(TARGET)/otp.mk diff --git a/lib/edoc/doc/overview.edoc b/lib/edoc/doc/overview.edoc index bd603b7a13..fa699c6f08 100644 --- a/lib/edoc/doc/overview.edoc +++ b/lib/edoc/doc/overview.edoc @@ -1084,10 +1084,11 @@ Details: the Erlang programming language.</li> <li>`boolean()' is the subset of `atom()' consisting of the atoms `true' and `false'.</li> - <li>`char()' is a subset of - `integer()' representing character codes.</li> + <li>`char()' is the subset of `integer()' representing + Unicode character codes: hex 000000-10FFFF.</li> <li>`tuple()' is the set of all tuples `{...}'.</li> - <li>`list(T)' is just an alias for `[T]'.</li> + <li>`list(T)' is just an alias for `[T]'; list() is an alias + for `list(any())', i.e., `[any()]'.</li> <li>`nil()' is an alias for the empty list `[]'.</li> <li>`cons(H,T)' is the list constructor. This is usually not used directly. It is possible to recursively define `list(T) diff --git a/lib/edoc/doc/src/Makefile b/lib/edoc/doc/src/Makefile index 5ee0096f0f..b933094464 100644 --- a/lib/edoc/doc/src/Makefile +++ b/lib/edoc/doc/src/Makefile @@ -13,8 +13,6 @@ # Portions created by Ericsson are Copyright 1999, Ericsson Utvecklings # AB. All Rights Reserved.'' # -# $Id$ -# include $(ERL_TOP)/make/target.mk include $(ERL_TOP)/make/$(TARGET)/otp.mk diff --git a/lib/edoc/include/Makefile b/lib/edoc/include/Makefile index 0533c27567..5b2ad38c9d 100644 --- a/lib/edoc/include/Makefile +++ b/lib/edoc/include/Makefile @@ -13,8 +13,6 @@ # Portions created by Ericsson are Copyright 1999, Ericsson Utvecklings # AB. All Rights Reserved.'' # -# $Id$ -# include $(ERL_TOP)/make/target.mk include $(ERL_TOP)/make/$(TARGET)/otp.mk diff --git a/lib/edoc/priv/edoc_generate.src b/lib/edoc/priv/edoc_generate.src index e87fdbc902..7ec89207b0 100644 --- a/lib/edoc/priv/edoc_generate.src +++ b/lib/edoc/priv/edoc_generate.src @@ -14,9 +14,6 @@ # Portions created by Ericsson are Copyright 1999-2000, Ericsson # Utvecklings AB. All Rights Reserved.'' # -# $Id$ -# -# #EDOC_DIR=/clearcase/otp/internal_tools/edoc EDOC_DIR=/home/otp/sgml/edoc-%EDOC_VSN% diff --git a/lib/edoc/src/Makefile b/lib/edoc/src/Makefile index 9c5a9d30d1..fcb0b61292 100644 --- a/lib/edoc/src/Makefile +++ b/lib/edoc/src/Makefile @@ -23,7 +23,7 @@ RELSYSDIR = $(RELEASE_PATH)/lib/edoc-$(VSN) EBIN = ../ebin XMERL = ../../xmerl -ERL_COMPILE_FLAGS += -I../include -I$(XMERL)/include +warn_unused_vars +nowarn_shadow_vars +warn_unused_import +warn_deprecated_guard +ERL_COMPILE_FLAGS += -pa $(XMERL) -I../include -I$(XMERL)/include +warn_unused_vars +nowarn_shadow_vars +warn_unused_import +warn_deprecated_guard SOURCES= \ edoc.erl edoc_data.erl edoc_doclet.erl edoc_extract.erl \ diff --git a/lib/edoc/src/edoc.erl b/lib/edoc/src/edoc.erl index 360f2dbc9e..a279f7dcb3 100644 --- a/lib/edoc/src/edoc.erl +++ b/lib/edoc/src/edoc.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @copyright 2001-2007 Richard Carlsson %% @author Richard Carlsson <[email protected]> %% @version {@version} @@ -60,8 +58,6 @@ -compile({no_auto_import,[error/1]}). --import(edoc_report, [report/2, report/3, error/1, error/3]). - -include("edoc.hrl"). @@ -179,8 +175,8 @@ application(App, Options) when is_atom(App) -> Dir when is_list(Dir) -> application(App, Dir, Options); _ -> - report("cannot find application directory for '~s'.", - [App]), + edoc_report:report("cannot find application directory for '~s'.", + [App]), exit(error) end. @@ -663,8 +659,8 @@ read_source(Name, Opts0) -> check_forms(Forms, Name), Forms; {error, R} -> - error({"error reading file '~s'.", - [edoc_lib:filename(Name)]}), + edoc_report:error({"error reading file '~s'.", + [edoc_lib:filename(Name)]}), exit({error, R}) end. @@ -688,11 +684,10 @@ check_forms(Fs, Name) -> error_marker -> case erl_syntax:error_marker_info(F) of {L, M, D} -> - error(L, Name, {format_error, M, D}); - + edoc_report:error(L, Name, {format_error, M, D}); Other -> - report(Name, "unknown error in " - "source code: ~w.", [Other]) + edoc_report:report(Name, "unknown error in " + "source code: ~w.", [Other]) end, exit(error); _ -> diff --git a/lib/edoc/src/edoc_data.erl b/lib/edoc/src/edoc_data.erl index 27f43dca5a..e3b5a0d51b 100644 --- a/lib/edoc/src/edoc_data.erl +++ b/lib/edoc/src/edoc_data.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @private %% @copyright 2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> diff --git a/lib/edoc/src/edoc_doclet.erl b/lib/edoc/src/edoc_doclet.erl index 30eef3e63a..c66be9d7c7 100644 --- a/lib/edoc/src/edoc_doclet.erl +++ b/lib/edoc/src/edoc_doclet.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @copyright 2003-2006 Richard Carlsson %% @author Richard Carlsson <[email protected]> %% @see edoc @@ -52,7 +50,7 @@ -define(IMAGE, "erlang.png"). -define(NL, "\n"). --include("xmerl.hrl"). +-include_lib("xmerl/include/xmerl.hrl"). %% Sources is the list of inputs in the order they were found. Packages %% and Modules are sorted lists of atoms without duplicates. (They diff --git a/lib/edoc/src/edoc_extract.erl b/lib/edoc/src/edoc_extract.erl index 5e28762c53..1209d86fe5 100644 --- a/lib/edoc/src/edoc_extract.erl +++ b/lib/edoc/src/edoc_extract.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id: $ -%% %% @copyright 2001-2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> %% @see edoc @@ -238,8 +236,8 @@ file(File, Context, Env, Opts) -> case file:read_file(File) of {ok, Bin} -> {ok, text(binary_to_list(Bin), Context, Env, Opts, File)}; - {error, _R} = Error -> - Error + {error, _} = Error -> + Error end. @@ -298,8 +296,8 @@ get_module_info(Forms, File) -> {Name, Vars} = case lists:keyfind(module, 1, L) of {module, N} when is_atom(N) -> {N, none}; - {module, {N, _Vs} = NVs} when is_atom(N) -> - NVs; + {module, {N, _}=Mod} when is_atom(N) -> + Mod; _ -> report(File, "module name missing.", []), exit(error) diff --git a/lib/edoc/src/edoc_layout.erl b/lib/edoc/src/edoc_layout.erl index 3ec87b7060..1c0841815f 100644 --- a/lib/edoc/src/edoc_layout.erl +++ b/lib/edoc/src/edoc_layout.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id: $ -%% %% @author Richard Carlsson <[email protected]> %% @copyright 2001-2006 Richard Carlsson %% @see edoc @@ -33,7 +31,7 @@ -import(edoc_report, [report/2]). --include("xmerl.hrl"). +-include_lib("xmerl/include/xmerl.hrl"). -define(HTML_EXPORT, xmerl_html). -define(DEFAULT_XML_EXPORT, ?HTML_EXPORT). @@ -959,12 +957,16 @@ local_label(R) -> xhtml(Title, CSS, Body) -> [{html, [?NL, - {head, [?NL, - {title, Title}, - ?NL] ++ CSS}, - ?NL, - {body, [{bgcolor, "white"}], Body}, - ?NL] + {head, [?NL, + {meta, [{'http-equiv',"Content-Type"}, + {content, "text/html; charset=ISO-8859-1"}], + []}, + ?NL, + {title, Title}, + ?NL] ++ CSS}, + ?NL, + {body, [{bgcolor, "white"}], Body}, + ?NL] }, ?NL]. diff --git a/lib/edoc/src/edoc_lib.erl b/lib/edoc/src/edoc_lib.erl index 585e30a2d2..6c698e83ef 100644 --- a/lib/edoc/src/edoc_lib.erl +++ b/lib/edoc/src/edoc_lib.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @copyright 2001-2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> %% @see edoc @@ -40,7 +38,7 @@ -import(edoc_report, [report/2, warning/2]). -include("edoc.hrl"). --include("xmerl.hrl"). +-include_lib("xmerl/include/xmerl.hrl"). -define(FILE_BASE, "/"). @@ -494,7 +492,7 @@ uri_get_file(File0) -> uri_get_http(URI) -> %% Try using option full_result=false case catch {ok, httpc:request(get, {URI,[]}, [], - [{full_result, false}])} of + [{full_result, false}])} of {'EXIT', _} -> uri_get_http_r10(URI); Result -> diff --git a/lib/edoc/src/edoc_parser.yrl b/lib/edoc/src/edoc_parser.yrl index 6943f1bdb8..3ce4cde4fb 100644 --- a/lib/edoc/src/edoc_parser.yrl +++ b/lib/edoc/src/edoc_parser.yrl @@ -23,9 +23,6 @@ %% USA %% %% Author contact: [email protected] -%% -%% $Id $ -%% %% ===================================================================== Nonterminals @@ -362,10 +359,10 @@ parse_spec(S, L) -> {ok, Spec} -> Spec; {error, E} -> - throw_error(E, L) + throw_error({parse_spec, E}, L) end; {error, E, _} -> - throw_error(E, L) + throw_error({parse_spec, E}, L) end. %% --------------------------------------------------------------------- diff --git a/lib/edoc/src/edoc_report.erl b/lib/edoc/src/edoc_report.erl index ee54c60c90..f082513bee 100644 --- a/lib/edoc/src/edoc_report.erl +++ b/lib/edoc/src/edoc_report.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @private %% @copyright 2001-2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> diff --git a/lib/edoc/src/edoc_run.erl b/lib/edoc/src/edoc_run.erl index 96e5ea4631..1355db840f 100644 --- a/lib/edoc/src/edoc_run.erl +++ b/lib/edoc/src/edoc_run.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @copyright 2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> %% @see edoc diff --git a/lib/edoc/src/edoc_scanner.erl b/lib/edoc/src/edoc_scanner.erl index 9d2e6f3aed..8e895ad1ad 100644 --- a/lib/edoc/src/edoc_scanner.erl +++ b/lib/edoc/src/edoc_scanner.erl @@ -13,8 +13,6 @@ %% AB. Portions created by Ericsson are Copyright 1999, Ericsson %% Utvecklings AB. All Rights Reserved.'' %% -%% $Id: $ -%% %% @private %% @copyright Richard Carlsson 2001-2003. Portions created by Ericsson %% are Copyright 1999, Ericsson Utvecklings AB. All Rights Reserved. diff --git a/lib/edoc/src/edoc_specs.erl b/lib/edoc/src/edoc_specs.erl index 79a5d142bc..bfb17515be 100644 --- a/lib/edoc/src/edoc_specs.erl +++ b/lib/edoc/src/edoc_specs.erl @@ -87,8 +87,9 @@ dummy_spec(Form) -> #tag{name = spec, line = element(2, hd(TypeSpecs)), origin = code, data = S}. --spec docs(Forms::[syntaxTree()], CommentFun) -> dict() when - CommentFun :: fun(([syntaxTree()], Line :: term()) -> #tag{}). +-spec docs(Forms::[syntaxTree()], + CommentFun :: fun( ([syntaxTree()], Line :: term()) -> #tag{} )) + -> dict(). %% @doc Find comments after -type/-opaque declarations. %% Postcomments "inside" the type are skipped. diff --git a/lib/edoc/src/edoc_tags.erl b/lib/edoc/src/edoc_tags.erl index 8ee8f87b5f..80989428ce 100644 --- a/lib/edoc/src/edoc_tags.erl +++ b/lib/edoc/src/edoc_tags.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @private %% @copyright 2001-2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> diff --git a/lib/edoc/src/edoc_types.erl b/lib/edoc/src/edoc_types.erl index e784b3359a..a54544868c 100644 --- a/lib/edoc/src/edoc_types.erl +++ b/lib/edoc/src/edoc_types.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @private %% @copyright 2001-2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> @@ -34,13 +32,13 @@ %% @headerfile "edoc_types.hrl" -include("edoc_types.hrl"). --include("xmerl.hrl"). +-include_lib("xmerl/include/xmerl.hrl"). is_predefined(any, 0) -> true; is_predefined(atom, 0) -> true; is_predefined(binary, 0) -> true; -is_predefined(bool, 0) -> true; +is_predefined(bool, 0) -> true; % kept for backwards compatibility is_predefined(char, 0) -> true; is_predefined(cons, 2) -> true; is_predefined(deep_string, 0) -> true; diff --git a/lib/edoc/src/edoc_wiki.erl b/lib/edoc/src/edoc_wiki.erl index ba33198787..2f2d14853c 100644 --- a/lib/edoc/src/edoc_wiki.erl +++ b/lib/edoc/src/edoc_wiki.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @private %% @copyright 2001-2003 Richard Carlsson %% @author Richard Carlsson <[email protected]> @@ -70,7 +68,7 @@ -export([parse_xml/2, expand_text/2]). -include("edoc.hrl"). --include("xmerl.hrl"). +-include_lib("xmerl/include/xmerl.hrl"). -define(BASE_HEADING, 3). @@ -82,8 +80,8 @@ parse_xml(Data, Line) -> parse_xml_1(Text, Line) -> Text1 = "<doc>" ++ Text ++ "</doc>", - Options = [{line, Line}, {encoding, "iso-8859-1"}], - case catch {ok, xmerl_scan:string(Text1, Options)} of + Opts = [{line, Line}, {encoding, 'iso-8859-1'}], + case catch {ok, xmerl_scan:string(Text1, Opts)} of {ok, {E, _}} -> E#xmlElement.content; {'EXIT', {fatal, {Reason, L, _C}}} -> diff --git a/lib/edoc/src/otpsgml_layout.erl b/lib/edoc/src/otpsgml_layout.erl index 45f74b299e..d425dc0ed8 100644 --- a/lib/edoc/src/otpsgml_layout.erl +++ b/lib/edoc/src/otpsgml_layout.erl @@ -14,8 +14,6 @@ %% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 %% USA %% -%% $Id$ -%% %% @author Richard Carlsson <[email protected]> %% @author Kenneth Lundin <[email protected]> %% @copyright 2001-2004 Richard Carlsson @@ -34,7 +32,7 @@ -import(edoc_report, [report/2]). --include("xmerl.hrl"). +-include_lib("xmerl/include/xmerl.hrl"). -define(SGML_EXPORT, xmerl_otpsgml). -define(DEFAULT_XML_EXPORT, ?SGML_EXPORT). diff --git a/lib/edoc/test/edoc_SUITE.erl b/lib/edoc/test/edoc_SUITE.erl index 0d57591e3e..5b95c35756 100644 --- a/lib/edoc/test/edoc_SUITE.erl +++ b/lib/edoc/test/edoc_SUITE.erl @@ -13,8 +13,6 @@ %% Portions created by Ericsson are Copyright 1999, Ericsson Utvecklings %% AB. All Rights Reserved.'' %% -%% $Id$ -%% -module(edoc_SUITE). -include_lib("test_server/include/test_server.hrl"). diff --git a/lib/et/src/et_wx_viewer.erl b/lib/et/src/et_wx_viewer.erl index 7d4286ed9d..386f8fc86b 100644 --- a/lib/et/src/et_wx_viewer.erl +++ b/lib/et/src/et_wx_viewer.erl @@ -257,10 +257,10 @@ parse_opt([H | T], S, CollectorOpt) -> Actors = [create_actor(Name) || Name <- ActorNames2], parse_opt(T, S#state{actors = Actors}, CollectorOpt); {include, ActorNames} when is_list(ActorNames) -> - Actors = [opt_create_actor(Name, include, S#state.actors) || Name <- ActorNames], + Actors = [opt_create_actor(Name, include, S) || Name <- ActorNames], parse_opt(T, S#state{actors = Actors}, CollectorOpt); {exclude, ActorNames} when is_list(ActorNames) -> - Actors = [opt_create_actor(Name, exclude, S#state.actors) || Name <- ActorNames], + Actors = [opt_create_actor(Name, exclude, S) || Name <- ActorNames], parse_opt(T, S#state{actors = Actors}, CollectorOpt); {first_event, _FirstKey} -> %% NYI diff --git a/lib/eunit/src/eunit_surefire.erl b/lib/eunit/src/eunit_surefire.erl index dfb08c90b2..25b5cde09c 100644 --- a/lib/eunit/src/eunit_surefire.erl +++ b/lib/eunit/src/eunit_surefire.erl @@ -64,6 +64,7 @@ }). -record(testsuite, { + id = 0 :: integer(), name = <<>> :: binary(), time = 0 :: integer(), output = <<>> :: binary(), @@ -76,7 +77,7 @@ -record(state, {verbose = false, indent = 0, xmldir = ".", - testsuite = #testsuite{} + testsuites = [] :: [#testsuite{}] }). start() -> @@ -89,55 +90,60 @@ init(Options) -> XMLDir = proplists:get_value(dir, Options, ?XMLDIR), St = #state{verbose = proplists:get_bool(verbose, Options), xmldir = XMLDir, - testsuite = #testsuite{}}, + testsuites = []}, receive {start, _Reference} -> St end. terminate({ok, _Data}, St) -> - TestSuite = St#state.testsuite, + TestSuites = St#state.testsuites, XmlDir = St#state.xmldir, - write_report(TestSuite, XmlDir), + write_reports(TestSuites, XmlDir), ok; terminate({error, _Reason}, _St) -> %% Don't report any errors here, since eunit_tty takes care of that. %% Just terminate. ok. -handle_begin(group, Data, St) -> +handle_begin(Kind, Data, St) when Kind == group; Kind == test -> + %% Run this code both for groups and tests; test is a bit + %% surprising: This is a workaround for the fact that we don't get + %% a group (handle_begin(group, ...) for testsuites (modules) + %% which only have one test case. In that case we get a test case + %% with an id comprised of just one integer - the group id. NewId = proplists:get_value(id, Data), case NewId of [] -> St; - [_GroupId] -> + [GroupId] -> Desc = proplists:get_value(desc, Data), - TestSuite = St#state.testsuite, - NewTestSuite = TestSuite#testsuite{name = Desc}, - St#state{testsuite=NewTestSuite}; + TestSuite = #testsuite{id = GroupId, name = Desc}, + St#state{testsuites=store_suite(TestSuite, St#state.testsuites)}; %% Surefire format is not hierarchic: Ignore subgroups: _ -> St - end; -handle_begin(test, _Data, St) -> - St. + end. handle_end(group, Data, St) -> %% Retrieve existing test suite: case proplists:get_value(id, Data) of [] -> St; - [_GroupId|_] -> - TestSuite = St#state.testsuite, + [GroupId|_] -> + TestSuites = St#state.testsuites, + TestSuite = lookup_suite_by_group_id(GroupId, TestSuites), %% Update TestSuite data: Time = proplists:get_value(time, Data), Output = proplists:get_value(output, Data), NewTestSuite = TestSuite#testsuite{ time = Time, output = Output }, - St#state{testsuite=NewTestSuite} + St#state{testsuites=store_suite(NewTestSuite, TestSuites)} end; handle_end(test, Data, St) -> %% Retrieve existing test suite: - TestSuite = St#state.testsuite, + [GroupId|_] = proplists:get_value(id, Data), + TestSuites = St#state.testsuites, + TestSuite = lookup_suite_by_group_id(GroupId, TestSuites), %% Create test case: Name = format_name(proplists:get_value(source, Data), @@ -149,7 +155,7 @@ handle_end(test, Data, St) -> TestCase = #testcase{name = Name, description = Desc, time = Time,output = Output}, NewTestSuite = add_testcase_to_testsuite(Result, TestCase, TestSuite), - St#state{testsuite=NewTestSuite}. + St#state{testsuites=store_suite(NewTestSuite, TestSuites)}. %% Cancel group does not give information on the individual cancelled test case %% We ignore this event @@ -157,7 +163,9 @@ handle_cancel(group, _Data, St) -> St; handle_cancel(test, Data, St) -> %% Retrieve existing test suite: - TestSuite = St#state.testsuite, + [GroupId|_] = proplists:get_value(id, Data), + TestSuites = St#state.testsuites, + TestSuite = lookup_suite_by_group_id(GroupId, TestSuites), %% Create test case: Name = format_name(proplists:get_value(source, Data), @@ -171,7 +179,7 @@ handle_cancel(test, Data, St) -> NewTestSuite = TestSuite#testsuite{ skipped = TestSuite#testsuite.skipped+1, testcases=[TestCase|TestSuite#testsuite.testcases] }, - St#state{testsuite=NewTestSuite}. + St#state{testsuites=store_suite(NewTestSuite, TestSuites)}. format_name({Module, Function, Arity}, Line) -> lists:flatten([atom_to_list(Module), ":", atom_to_list(Function), "/", @@ -183,6 +191,12 @@ format_desc(Desc) when is_binary(Desc) -> format_desc(Desc) when is_list(Desc) -> Desc. +lookup_suite_by_group_id(GroupId, TestSuites) -> + #testsuite{} = lists:keyfind(GroupId, #testsuite.id, TestSuites). + +store_suite(#testsuite{id=GroupId} = TestSuite, TestSuites) -> + lists:keystore(GroupId, #testsuite.id, TestSuites, TestSuite). + %% Add testcase to testsuite depending on the result of the test. add_testcase_to_testsuite(ok, TestCaseTmp, TestSuite) -> TestCase = TestCaseTmp#testcase{ result = ok }, @@ -214,6 +228,10 @@ add_testcase_to_testsuite({error, Exception}, TestCaseTmp, TestSuite) -> %% Write a report to the XML directory. %% This function opens the report file, calls write_report_to/2 and closes the file. %% ---------------------------------------------------------------------------- +write_reports(TestSuites, XmlDir) -> + lists:foreach(fun(TestSuite) -> write_report(TestSuite, XmlDir) end, + TestSuites). + write_report(#testsuite{name = Name} = TestSuite, XmlDir) -> Filename = filename:join(XmlDir, lists:flatten(["TEST-", escape_suitename(Name)], ".xml")), case file:open(Filename, [write, raw]) of diff --git a/lib/ic/doc/src/notes.xml b/lib/ic/doc/src/notes.xml index 5f6c31069c..de519d5f84 100644 --- a/lib/ic/doc/src/notes.xml +++ b/lib/ic/doc/src/notes.xml @@ -4,7 +4,7 @@ <chapter> <header> <copyright> - <year>1998</year><year>2010</year> + <year>1998</year><year>2011</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -31,6 +31,22 @@ </header> <section> + <title>IC 4.2.27</title> + + <section> + <title>Improvements and New Features</title> + <list type="bulleted"> + <item> + <p> + Reduced compile overhead (Thanks to Haitao Li).</p> + <p> + Own Id: OTP-9460 </p> + </item> + </list> + </section> + </section> + + <section> <title>IC 4.2.26</title> <section> diff --git a/lib/ic/src/ic_pp.erl b/lib/ic/src/ic_pp.erl index db06118d32..8b53473caa 100644 --- a/lib/ic/src/ic_pp.erl +++ b/lib/ic/src/ic_pp.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1997-2009. All Rights Reserved. +%% Copyright Ericsson AB 1997-2011. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -92,6 +92,14 @@ %% %%====================================================================================== +%% Multiple Include Optimization +%% +%% Algorithm described at: +%% http://gcc.gnu.org/onlinedocs/cppinternals/Guard-Macros.html +-record(mio, {valid = true, %% multiple include valid + cmacro, %% controlling macro of the current conditional directive + depth = 0, %% conditional directive depth + included = []}). @@ -130,7 +138,7 @@ run(FileList, FileName, IncDir, Flags) -> %%---------------------------------------------------------- %% Run the second phase, i.e expand macros %%---------------------------------------------------------- - {Out, Err, War, _Defs, IfCou} = expand(File, FileName, IncDir, Flags), + {Out, Err, War, _Defs, _Mio, IfCou} = expand(File, FileName, IncDir, Flags), %%---------------------------------------------------------- %% Check if all #if #ifdef #ifndef have a matching #endif @@ -155,9 +163,9 @@ run(FileList, FileName, IncDir, Flags) -> %% The entry for all included files %% %% -%% Output {Out, Defs, Err, War} +%% Output {Out, Err, War, Defs, MultipleIncludeValid} %%====================================================================================== -run_include(FileName, FileList, _Out, Defs, Err, War, IncLine, IncFile, IncDir) -> +run_include(FileName, FileList, _Out, Defs, Err, War, IncLine, IncFile, IncDir, Mio) -> %%---------------------------------------------------------- %% Run the first phase, i.e tokenise the file @@ -169,18 +177,21 @@ run_include(FileName, FileList, _Out, Defs, Err, War, IncLine, IncFile, IncDir) %%---------------------------------------------------------- %% Run the second phase, i.e expand macros %%---------------------------------------------------------- - - %% Try first pass without file info start/end - {OutT, ErrT, WarT, DefsT, IfCouT} = - expand(File, Defs, Err, War, [FileName|IncFile], IncDir), - - {Out2, Err2, War2, Defs2, IfCou2} = - case only_nls(OutT) of - true -> %% The file is defined before - {["\n"], ErrT, WarT, DefsT, IfCouT}; - false -> %% The file is not defined before, try second pass - expand([FileInfoStart|File]++FileInfoEnd, Defs, Err, War, [FileName|IncFile], IncDir) - end, + {Out2, Err2, War2, Defs2, Mio2, IfCou2} = + expand([FileInfoStart|File]++FileInfoEnd, Defs, Err, War, + [FileName|IncFile], IncDir, + #mio{included=Mio#mio.included}), + + MergeIncluded = sets:to_list(sets:from_list(Mio#mio.included ++ Mio2#mio.included)), + + Mio3 = + case {Mio2#mio.valid, Mio2#mio.cmacro} of + {V, Macro} when V == false; + Macro == undefined -> + update_mio(Mio#mio{included=MergeIncluded}); + {true, _} -> + update_mio({include, FileName}, Mio#mio{included=MergeIncluded}) + end, %%---------------------------------------------------------- %% Check if all #if #ifdef #ifndef have a matching #endif @@ -192,26 +203,7 @@ run_include(FileName, FileList, _Out, Defs, Err, War, IncLine, IncFile, IncDir) [] end, - {Out2, Defs2, Err2++IfError, War2}. - - - -%% Return true if there is no data -%% other than new lines -only_nls([]) -> - true; -only_nls(["\n"|Rem]) -> - only_nls(Rem); -only_nls(["\r","\n"|Rem]) -> - only_nls(Rem); -only_nls([_|_Rem]) -> - false. - - - - - - + {Out2, Defs2, Err2++IfError, War2, Mio3}. @@ -647,87 +639,86 @@ expand(List, FileName, IncDir, Flags) -> %% Get all definitions from preprocessor commnads %% and merge them on top of the file collected. CLDefs = get_cmd_line_defs(Flags), - expand(List, [], [], CLDefs, [FileName], IncDir, check_all, [], [], 1, FileName). - -expand(List, Defs, Err, War, [FileName|IncFile], IncDir) -> - expand(List, [], [], Defs, [FileName|IncFile], IncDir, check_all, Err, War, 1, FileName). + expand(List, [], [], CLDefs, [FileName], IncDir, #mio{}, check_all, [], [], 1, FileName). +expand(List, Defs, Err, War, [FileName|IncFile], IncDir, Mio) -> + expand(List, [], [], Defs, [FileName|IncFile], IncDir, Mio, check_all, Err, War, 1, FileName). %%======================================================= %% Main loop for the expansion %%======================================================= -expand([], Out, _SelfRef, Defs, _IncFile, _IncDir, IfCou, Err, War, _L, _FN) -> +expand([], Out, _SelfRef, Defs, _IncFile, _IncDir, Mio, IfCou, Err, War, _L, _FN) -> % io:format("~n ===============~n"), % io:format(" definitions ~p~n",[lists:reverse(Defs)]), % io:format(" found warnings ~p~n",[lists:reverse(War)]), % io:format(" found errors ~p~n",[lists:reverse(Err)]), % io:format(" ===============~n~n~n"), - {Out, Err, War, Defs, IfCou}; + {Out, Err, War, Defs, Mio, IfCou}; -expand([{file_info, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, Str++Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([{file_info, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, Str++Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN); %%--------------------------------------- %% Searching for endif, %% i.e skip all source lines until matching %% end if is encountered %%--------------------------------------- -expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) +expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) when Command == "ifdef" -> {_Removed, Rem2, _Nl} = read_to_nl(Rem), IfCou2 = {endif, Endif+1, IfLine}, - expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, IfCou2, Err, War, L, FN); + expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err, War, L, FN); -expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) +expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) when Command == "ifndef" -> {_Removed, Rem2, _Nl} = read_to_nl(Rem), IfCou2 = {endif, Endif+1, IfLine}, - expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, IfCou2, Err, War, L, FN); + expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err, War, L, FN); -expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) +expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) when Command == "if" -> - case pp_command(Command, Rem, Defs, IncDir, Err, War, L, FN) of + case pp_command(Command, Rem, Defs, IncDir, Mio, Err, War, L, FN) of {{'if', true}, Rem2, Err2, War2, Nl} -> IfCou2 = {endif, Endif+1, IfLine}, - expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, IfCou2, Err2, War2, L+Nl, FN); + expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err2, War2, L+Nl, FN); %% {{'if', false}, Rem2, Err2, War2, Nl} -> Not implemented yet {{'if', error}, Rem2, Err2, War2, Nl} -> IfCou2 = {endif, Endif, IfLine}, - expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, IfCou2, Err2, War2, L+Nl, FN) + expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err2, War2, L+Nl, FN) end; -expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) +expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) when Command == "endif" -> {_Removed, Rem2, Nl} = read_to_nl(Rem), case Endif of 1 -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio, check_all, Err, War, L+Nl, FN); _ -> IfCou2 = {endif, Endif-1, IfLine}, - expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, IfCou2, Err, War, L+Nl, FN) + expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err, War, L+Nl, FN) end; -expand([{command,_Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) -> +expand([{command,_Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) -> {_Removed, Rem2, _Nl} = read_to_nl(Rem), IfCou2 = {endif, Endif, IfLine}, - expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, IfCou2, Err, War, L, FN); + expand(Rem2, Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err, War, L, FN); %% Solves a bug when spaces in front of hashmark ! -expand([space | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) -> - expand(Rem, Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN); +expand([space | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) -> + expand(Rem, Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN); -expand([{nl,_Nl} | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) -> - expand(Rem, Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN); +expand([{nl,_Nl} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) -> + expand(Rem, Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN); -expand([_X | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN) -> +expand([_X | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN) -> {_Removed, Rem2, Nl} = read_to_nl(Rem), - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, Err, War, L, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio, {endif, Endif, IfLine}, Err, War, L, FN); @@ -736,121 +727,132 @@ expand([_X | Rem], Out, SelfRef, Defs, IncFile, IncDir, {endif, Endif, IfLine}, %%--------------------------------------- %% Check all tokens %%--------------------------------------- -expand([{nl, _N} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [$\n | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L+1, FN); +expand([{nl, _N} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [$\n | Out], SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L+1, FN); -expand([space | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [?space | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([space | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [?space | Out], SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN); -expand([space_exp | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [?space | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([space_exp | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [?space | Out], SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN); -expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L, FN) -> - case pp_command(Command, Rem, Defs, IncDir, Err, War, L, FN) of +expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, check_all, Err, War, L, FN) -> + case pp_command(Command, Rem, Defs, IncDir, Mio, Err, War, L, FN) of {define, Rem2, Defs2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs2, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs2, IncFile, IncDir, update_mio(Mio), check_all, Err2, War2, L+Nl, FN); {undef, Rem2, Defs2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs2, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs2, IncFile, IncDir, update_mio(Mio), check_all, Err2, War2, L+Nl, FN); {{include, ok}, FileName, FileCont, Rem2, Nl, Err2, War2} -> - {Out3, Defs3, Err3, War3} = - run_include(FileName, FileCont, Out, Defs, Err2, War2, L+Nl, IncFile, IncDir), + {Out3, Defs3, Err3, War3, Mio2} = + run_include(FileName, FileCont, Out, Defs, Err2, War2, L+Nl, IncFile, IncDir, Mio), Nls = [], Out4 = Out3++Nls++Out, - expand(Rem2, Out4, SelfRef, Defs3, IncFile, IncDir, check_all, Err3, War3, L+Nl, FN); + expand(Rem2, Out4, SelfRef, Defs3, IncFile, IncDir, Mio2, check_all, Err3, War3, L+Nl, FN); {{include, error}, Rem2, Nl, Err2, War2} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, Err2, War2, L+Nl, FN); + + {{include, skip}, Rem2} -> + Out2 = [$\n|Out], + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, Err, War, L+1, FN); {{ifdef, true}, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], + Out2 = lists:duplicate(Nl,$\n) ++ Out, IfCou2 = {endif, 1, L}, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, IfCou2, Err2, War2, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err2, War2, L+Nl, FN); {{ifdef, false}, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + Mio2 = update_mio(ifdef, Mio), + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err2, War2, L+Nl, FN); {{ifndef, true}, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], + Out2 = lists:duplicate(Nl,$\n) ++ Out, IfCou2 = {endif, 1, L}, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, IfCou2, Err2, War2, L+Nl, FN); - {{ifndef, false}, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err2, War2, L+Nl, FN); + {{ifndef, false}, Macro, Rem2, Err2, War2, Nl} -> + Out2 = lists:duplicate(Nl,$\n) ++ Out, + Mio2 = update_mio({ifndef, Macro}, Mio), + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err2, War2, L+Nl, FN); {endif, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + Mio2 = update_mio(endif, Mio), + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err2, War2, L+Nl, FN); {{'if', true}, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], + Out2 = lists:duplicate(Nl,$\n) ++ Out, IfCou2 = {endif, 1, L}, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, IfCou2, Err2, War2, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio, IfCou2, Err2, War2, L+Nl, FN); %% {{'if', false}, Removed, Rem2, Nl} -> Not implemented at present {{'if', error}, Rem2, Err2, War2, Nl} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err2, War2, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + Mio2 = update_mio('if', Mio), + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err2, War2, L+Nl, FN); {'else', {_Removed, Rem2, Nl}} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], + Out2 = lists:duplicate(Nl,$\n) ++ Out, Err2 = {FN, L, "`else' command is not implemented at present"}, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, [Err2|Err], War, L+Nl, FN); + Mio2 = update_mio('else', Mio), + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, [Err2|Err], War, L+Nl, FN); {'elif', {_Removed, Rem2, Nl}} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], + Out2 = lists:duplicate(Nl,$\n) ++ Out, Err2 = {FN, L, "`elif' command is not implemented at present"}, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, [Err2|Err], War, L+Nl, FN); + Mio2 = update_mio('elif', Mio), + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, [Err2|Err], War, L+Nl, FN); {warning, {WarningText, Rem2, Nl}} -> [FileName|_More] = IncFile, War2 = {FileName, L, "warning: #warning "++detokenise(WarningText)}, - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, [War2|War], L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, Err, [War2|War], L+Nl, FN); {error, {ErrorText, Rem2, Nl}} -> [FileName|_More] = IncFile, Err2 = {FileName, L, detokenise(ErrorText)}, - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, [Err2|Err], War, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, [Err2|Err], War, L+Nl, FN); {{line, ok}, {_Removed, Rem2, Nl}, L2, FN2, LineText} -> Out2 = lists:duplicate(Nl,$\n)++LineText++Out, [_X|IF] = IncFile, IncFile2 = [FN2|IF], - expand(Rem2, Out2, SelfRef, Defs, IncFile2, IncDir, check_all, Err, War, L2, FN2); + expand(Rem2, Out2, SelfRef, Defs, IncFile2, IncDir, update_mio(Mio), check_all, Err, War, L2, FN2); {{line, error}, {_Removed, Rem2, Nl}, Err2} -> - Out2 = [lists:duplicate(Nl,$\n)|Out], - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, [Err2|Err], War, L+Nl, FN); + Out2 = lists:duplicate(Nl,$\n) ++ Out, + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, [Err2|Err], War, L+Nl, FN); hash_mark -> - expand(Rem, Out, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L, FN); + expand(Rem, Out, SelfRef, Defs, IncFile, IncDir, Mio, check_all, Err, War, L, FN); {pragma, Rem2, Nl, Text} -> Out2 = lists:duplicate(Nl,$\n)++Text++Out, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, Err, War, L+Nl, FN); {ident, Rem2, Nl, Text} -> Out2 = lists:duplicate(Nl,$\n)++Text++Out, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, update_mio(Mio), check_all, Err, War, L+Nl, FN); {not_recognised, {Removed, Rem2, Nl}} -> Text = lists:reverse([$#|Command]), RemovedS = lists:reverse([?space|detokenise(Removed)]), Out2 = [$\n|RemovedS]++Text++Out, + Mio2 = update_mio(Mio), case Command of [X|_T] when ?is_upper(X) -> - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err, War, L+Nl, FN); [X|_T] when ?is_lower(X) -> - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err, War, L+Nl, FN); [X|_T] when ?is_underline(X) -> - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, Err, War, L+Nl, FN); + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, Err, War, L+Nl, FN); _ -> Err2 = {FN, L, "invalid preprocessing directive name"}, - expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, check_all, [Err2|Err], War, L+Nl, FN) + expand(Rem2, Out2, SelfRef, Defs, IncFile, IncDir, Mio2, check_all, [Err2|Err], War, L+Nl, FN) end; Else -> @@ -859,19 +861,19 @@ expand([{command,Command} | Rem], Out, SelfRef, Defs, IncFile, IncDir, check_all end; -expand([{var, "__LINE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{var, "__LINE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> LL = io_lib:format("~p",[L]), - expand(Rem, [LL | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); + expand(Rem, [LL | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{var, "__FILE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [$",FN,$" | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([{var, "__FILE__"}|Rem], Out, SelfRef, Defs, IncFile, Mio, IncDir, IfCou, Err, War, L, FN) -> + expand(Rem, [$",FN,$" | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{var, "__DATE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{var, "__DATE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> {{Y,M,D},{_H,_Mi,_S}} = calendar:universal_time(), Date = io_lib:format("\"~s ~p ~p\"",[month(M),D,Y]), - expand(Rem, [Date | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); + expand(Rem, [Date | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{var, "__TIME__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{var, "__TIME__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> {{_Y,_M,_D},{H,Mi,S}} = calendar:universal_time(), HS = if H < 10 -> "0"++integer_to_list(H); true -> integer_to_list(H) @@ -883,40 +885,40 @@ expand([{var, "__TIME__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, true -> integer_to_list(S) end, Time = io_lib:format("\"~s:~s:~s\"",[HS,MiS,SS]), - expand(Rem, [Time | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); + expand(Rem, [Time | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{var, "__INCLUDE_LEVEL__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{var, "__INCLUDE_LEVEL__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> IL = io_lib:format("~p",[length(IncFile)-1]), - expand(Rem, [IL | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); + expand(Rem, [IL | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{var, "__BASE_FILE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{var, "__BASE_FILE__"}|Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> [BF|_T] = lists:reverse(IncFile), - expand(Rem, [$",BF,$" | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); + expand(Rem, [$",BF,$" | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{var, Var} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{var, Var} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> {Out2, Err2, War2, Rem2, SelfRef2} = source_line(Var, Rem, SelfRef, Defs, Err, War, L, FN), - expand(Rem2, [Out2 | Out], SelfRef2, Defs, IncFile, IncDir, IfCou, Err2, War2, L, FN); + expand(Rem2, [Out2 | Out], SelfRef2, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err2, War2, L, FN); -expand([{char, Char} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [Char | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([{char, Char} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [Char | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{number, Number} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [Number | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([{number, Number} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [Number | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{expanded, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [Str | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([{expanded, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [Str | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{self_ref, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{self_ref, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> SelfRef2 = lists:delete(Str,SelfRef), - expand(Rem, Out, SelfRef2, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); + expand(Rem, Out, SelfRef2, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{string, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> - expand(Rem, [$", Str, $" | Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN); +expand([{string, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> + expand(Rem, [$", Str, $" | Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L, FN); -expand([{string_part, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L, FN) -> +expand([{string_part, Str} | Rem], Out, SelfRef, Defs, IncFile, IncDir, Mio, IfCou, Err, War, L, FN) -> {Str2, Rem2, Nl} = expand_string_part([$"|Str], Rem), - expand(Rem2, [Str2| Out], SelfRef, Defs, IncFile, IncDir, IfCou, Err, War, L+Nl, FN). + expand(Rem2, [Str2| Out], SelfRef, Defs, IncFile, IncDir, update_mio(Mio), IfCou, Err, War, L+Nl, FN). @@ -954,13 +956,14 @@ expand_string_part([{string_part, Str_part} | Rem], Str, Nl) -> get_cmd_line_defs(Flags) -> Adjusted = parse_cmd_line(Flags,[]), - {_Out, _Err, _War, Defs, _IfCou} = + {_Out, _Err, _War, Defs, _IfCou, _Mio} = expand(tokenise(Adjusted,""), [], [], [], [], [], + #mio{}, check_all, [], [], @@ -1030,10 +1033,10 @@ collect_undefine([C|Rest],Found) -> %%====================================================================================== %%====================================================================================== -pp_command(Command, [space|File], Defs, IncDir, Err, War, L, FN) -> - pp_command(Command, File, Defs, IncDir, Err, War, L, FN); +pp_command(Command, [space|File], Defs, IncDir, Mio, Err, War, L, FN) -> + pp_command(Command, File, Defs, IncDir, Mio, Err, War, L, FN); -pp_command(Command, File, Defs, IncDir, Err, War, L, FN) -> +pp_command(Command, File, Defs, IncDir, Mio, Err, War, L, FN) -> case Command of %%---------------------------------------- @@ -1081,14 +1084,16 @@ pp_command(Command, File, Defs, IncDir, Err, War, L, FN) -> %% #include %%---------------------------------------- "include" -> - case include(File, IncDir) of - {error, Rem, Nl, Err2} -> - {{include, error}, Rem, Nl, [{FN, L, Err2}|Err], War}; - {error, Rem, Nl, Err2, NameNl} -> - {{include, error}, Rem, Nl, [{FN, L+ NameNl, Err2}|Err], War}; - {ok, FileName, FileCont, Rem, Nl} -> - {{include, ok}, FileName, FileCont, Rem, Nl, Err, War} - end; + case include(File, IncDir, Mio) of + {error, Rem, Nl, Err2} -> + {{include, error}, Rem, Nl, [{FN, L, Err2}|Err], War}; + {error, Rem, Nl, Err2, NameNl} -> + {{include, error}, Rem, Nl, [{FN, L+ NameNl, Err2}|Err], War}; + {ok, FileNamePath, FileCont, Rem, Nl} -> + {{include, ok}, FileNamePath, FileCont, Rem, Nl, Err, War}; + {skip, Rem} -> + {{include, skip}, Rem} + end; %%---------------------------------------- %% #ifdef @@ -1127,14 +1132,14 @@ pp_command(Command, File, Defs, IncDir, Err, War, L, FN) -> yes -> {{ifndef, true}, Rem, Err2, War2, Nl}; no -> - {{ifndef, false}, Rem, Err2, War2, Nl} + {{ifndef, false}, Name, Rem, Err2, War2, Nl} end; {ok, Rem, Name, No_of_para, _Parameters, _Macro, Err2, War2, Nl} -> case is_defined_before(Name, No_of_para, Defs) of yes -> {{ifndef, true}, Rem, Err2, War2, Nl}; no -> - {{ifndef, false}, Rem, Err2, War2, Nl} + {{ifndef, false}, Name, Rem, Err2, War2, Nl} end end; @@ -1408,29 +1413,32 @@ undef(_Rem) -> %%=============================================================== %%=============================================================== -include(File, IncDir) -> +include(File, IncDir, Mio) -> case include2(File) of - {ok, FileName, Rem, Nl, FileType} -> - %% The error handling is lite strange just to make it compatible to gcc - case {read_inc_file(FileName, IncDir), Nl, FileType} of - {{ok, FileList, FileNamePath}, _, _} -> - {ok, FileNamePath, FileList, Rem, Nl}; - {{error, Text}, _, own_file} -> - NameNl = count_nl(FileName,0), - Error = lists:flatten(io_lib:format("~s: ~s",[FileName,Text])), - {error, Rem, Nl, Error, NameNl}; - {{error, Text}, 1, sys_file} -> - NameNl = count_nl(FileName,0), - Error = lists:flatten(io_lib:format("~s: ~s",[FileName,Text])), - {error, Rem, Nl, Error, NameNl}; - {{error, _Text}, _, sys_file} -> - {error, Rem, Nl, "`#include' expects \"FILENAME\" or <FILENAME>"} - end; - - {error, {_Removed, Rem, Nl}} -> - {error, Rem, Nl, "`#include' expects \"FILENAME\" or <FILENAME>"} + {ok, FileName, Rem, Nl, FileType} -> + Result = read_inc_file(FileName, IncDir, Mio), + case {Result, Nl, FileType} of + {{ok, FileNamePath, FileCont}, _, _} -> + {ok, FileNamePath, FileCont, Rem, Nl}; + {skip, _, _} -> + {skip, Rem}; + {{error, Text}, _, own_file} -> + NameNl = count_nl(FileName,0), + Error = lists:flatten(io_lib:format("~s: ~s",[FileName,Text])), + {error, Rem, Nl, Error, NameNl}; + {{error, Text}, 1, sys_file} -> + NameNl = count_nl(FileName,0), + Error = lists:flatten(io_lib:format("~s: ~s",[FileName,Text])), + {error, Rem, Nl, Error, NameNl}; + {{error, _Text}, _, sys_file} -> + {error, Rem, Nl, "`#include' expects \"FILENAME\" or <FILENAME>"} + end; + {error, {_Removed, Rem, Nl}} -> + {error, Rem, Nl, "`#include' expects \#FILENAME\" or <FILENAME>"} end. + + count_nl([],Nl) -> Nl; count_nl([$\n|T],Nl) -> @@ -1909,18 +1917,37 @@ include_dir(Flags,IncDir) -> %% Read a included file. Try current dir first then the IncDir list %%=============================================================== -read_inc_file(FileName, IncDir) -> - case catch file:read_file(FileName) of - {ok, Bin} -> - FileList = binary_to_list(Bin), - {ok, FileList, FileName}; +read_inc_file(FileName, IncDir, Mio) -> + case find_inc_file(FileName, IncDir) of + {ok, AbsFile} -> + %% is included before? + case lists:member(FileName, Mio#mio.included) of + false -> + case catch file:read_file(AbsFile) of + {ok, Bin} -> + FileList = binary_to_list(Bin), + {ok, AbsFile, FileList}; + {error, Text} -> + {error, Text} + end; + true -> + skip + end; + {error, Text} -> + {error, Text} + end. + +find_inc_file(FileName, IncDir) -> + case catch file:read_file_info(FileName) of + {ok, _} -> + {ok, FileName}; {error, _} -> - read_inc_file2(FileName, IncDir) + find_inc_file2(FileName, IncDir) end. -read_inc_file2(_FileName, []) -> +find_inc_file2(_FileName, []) -> {error, "No such file or directory"}; -read_inc_file2(FileName, [D|Rem]) -> +find_inc_file2(FileName, [D|Rem]) -> Dir = case lists:last(D) of $/ -> D; @@ -1928,17 +1955,14 @@ read_inc_file2(FileName, [D|Rem]) -> D++"/" end, - case catch file:read_file(Dir++FileName) of - {ok, Bin} -> - FileList = binary_to_list(Bin), - {ok, FileList, Dir++FileName}; + case catch file:read_file_info(Dir++FileName) of + {ok, _} -> + {ok, Dir++FileName}; {error, _} -> - read_inc_file2(FileName, Rem) + find_inc_file2(FileName, Rem) end. - - %%=============================================================== %% Read parameters of a macro or a variable in a source line %%=============================================================== @@ -2135,5 +2159,73 @@ month(11) -> "Nov"; month(12) -> "Dec". +%% Multiple Include Optimization +%% +%% Algorithm described at: +%% http://gcc.gnu.org/onlinedocs/cppinternals/Guard-Macros.html +update_mio({include, FileName}, #mio{included=Inc}=Mio) -> + Mio#mio{valid=false, included=[FileName|Inc]}; + +%% valid=false & cmacro=undefined indicates it is already decided this file is +%% not subject to MIO +update_mio(_, #mio{valid=false, depth=0, cmacro=undefined}=Mio) -> + Mio; + +%% if valid=true, there is no non-whitespace tokens before this ifndef +update_mio({'ifndef', Macro}, #mio{valid=true, depth=0, cmacro=undefined}=Mio) -> + Mio#mio{valid=false, cmacro=Macro, depth=1}; + +%% detect any tokens before top level #ifndef +update_mio(_, #mio{valid=true, depth=0, cmacro=undefined}=Mio) -> + Mio#mio{valid=false}; + +%% If cmacro is alreay set, this is after the top level #endif +update_mio({'ifndef', _}, #mio{valid=true, depth=0}=Mio) -> + Mio#mio{valid=false, cmacro=undefined}; + +%% non-top level conditional, just update depth +update_mio({'ifndef', _}, #mio{depth=D}=Mio) when D > 0 -> + Mio#mio{depth=D+1}; +update_mio('ifdef', #mio{depth=D}=Mio) -> + Mio#mio{depth=D+1}; +update_mio('if', #mio{depth=D}=Mio) -> + Mio#mio{depth=D+1}; + +%% top level #else #elif invalidates multiple include optimization +update_mio('else', #mio{depth=1}=Mio) -> + Mio#mio{valid=false, cmacro=undefined}; +update_mio('else', Mio) -> + Mio; +update_mio('elif', #mio{depth=1}=Mio) -> + Mio#mio{valid=false, cmacro=undefined}; +update_mio('elif', Mio) -> + Mio; + +%% AT exit to top level, if the controlling macro is not set, this could be the +%% end of a non-ifndef conditional block, or there were tokens before entering +%% the #ifndef block. In either way, this invalidates the MIO +%% +%% It doesn't matter if `valid` is true at the time of exiting, it is set to +%% true. This will be used to detect if more tokens are following the top +%% level #endif. +update_mio('endif', #mio{depth=1, cmacro=undefined}=Mio) -> + Mio#mio{valid=false, depth=0}; +update_mio('endif', #mio{depth=1}=Mio) -> + Mio#mio{valid=true, depth=0}; +update_mio('endif', #mio{depth=D}=Mio) when D > 1 -> + Mio#mio{valid=true, depth=D-1}; + +%%if more tokens are following the top level #endif. +update_mio('endif', #mio{depth=1, cmacro=undefined}=Mio) -> + Mio#mio{valid=false, depth=0}; +update_mio('endif', #mio{depth=D}=Mio) when D > 0 -> + Mio#mio{valid=true, depth=D-1}; +update_mio(_, Mio) -> + Mio#mio{valid=false}. + +%% clear `valid`, this doesn't matter since #endif will restore it if +%% appropriate +update_mio(Mio) -> + Mio#mio{valid=false}. diff --git a/lib/ic/src/ic_pragma.erl b/lib/ic/src/ic_pragma.erl index 45cb64c9c8..7f2216b9dc 100644 --- a/lib/ic/src/ic_pragma.erl +++ b/lib/ic/src/ic_pragma.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1998-2010. All Rights Reserved. +%% Copyright Ericsson AB 1998-2011. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -1600,9 +1600,8 @@ remove_inheriters(S,RS,InheriterList) -> [_OneOnly] -> ReducedInhList; _Other -> - EtsList = ets:tab2list(S), CleanList = - [X || X <- EtsList, element(1,X) == inherits], + ets:match(S, {inherits,'_','_'}), % CodeOptList = % [X || X <- EtsList, element(1,X) == codeopt], NoInheriters =remove_inheriters2(S,ReducedInhList,CleanList), @@ -1648,9 +1647,8 @@ remove_inh([X],[Y],List,EtsList) -> %%% from others in the list %%%---------------------------------------------- remove_inherited(S,InheriterList) -> - EtsList = ets:tab2list(S), CleanList = - [X || X <- EtsList, element(1,X) == inherits], + ets:match(S, {inherits, '_', '_'}), remove_inherited(S,InheriterList,CleanList). @@ -1694,11 +1692,8 @@ remove_inhed([X],[Y],List,EtsList) -> %% are inherited from scope in the list %%%---------------------------------------------- get_inherited(S,Scope,OpScopeList) -> - EtsList = ets:tab2list(S), - [[element(3,X)] || X <- EtsList, - element(1,X) == inherits, - element(2,X) == Scope, - member([element(3,X)],OpScopeList)]. + EtsList1 = ets:match(S, {inherits, Scope, '$1'}), + [X || X <- EtsList1, member(X, OpScopeList)]. @@ -1771,9 +1766,7 @@ inherits2(_X,Y,Z,EtsList) -> %% false otherwise %% is_inherited_by(Interface1,Interface2,PragmaTab) -> - FullList = ets:tab2list(PragmaTab), - InheritsList = - [X || X <- FullList, element(1,X) == inherits], + InheritsList = ets:match(PragmaTab, {inherits, '_', '_'}), inherits(Interface2,Interface1,InheritsList). diff --git a/lib/ic/vsn.mk b/lib/ic/vsn.mk index 6d6c7fa625..6561ccd2a7 100644 --- a/lib/ic/vsn.mk +++ b/lib/ic/vsn.mk @@ -1 +1 @@ -IC_VSN = 4.2.26 +IC_VSN = 4.2.27 diff --git a/lib/kernel/doc/src/gen_sctp.xml b/lib/kernel/doc/src/gen_sctp.xml index 5ceb82ae41..b761b6bd83 100644 --- a/lib/kernel/doc/src/gen_sctp.xml +++ b/lib/kernel/doc/src/gen_sctp.xml @@ -990,7 +990,7 @@ server(IP, Port) when is_tuple(IP) orelse IP == any orelse IP == loopback, is_integer(Port) -> - {ok,S} = gen_sctp:open([{ip,IP},{port,Port}],[{recbuf,65536}]), + {ok,S} = gen_sctp:open(Port, [{recbuf,65536}, {ip,IP}]), io:format("Listening on ~w:~w. ~w~n", [IP,Port,S]), ok = gen_sctp:listen(S, true), server_loop(S). diff --git a/lib/kernel/doc/src/gen_tcp.xml b/lib/kernel/doc/src/gen_tcp.xml index f1d42d9faa..ebd822359a 100644 --- a/lib/kernel/doc/src/gen_tcp.xml +++ b/lib/kernel/doc/src/gen_tcp.xml @@ -37,7 +37,7 @@ binary and closing the connection:</p> <code type="none"> client() -> - SomeHostInNet = "localhost" % to make it runnable on one machine + SomeHostInNet = "localhost", % to make it runnable on one machine {ok, Sock} = gen_tcp:connect(SomeHostInNet, 5678, [binary, {packet, 0}]), ok = gen_tcp:send(Sock, "Some Data"), diff --git a/lib/kernel/test/gen_udp_SUITE.erl b/lib/kernel/test/gen_udp_SUITE.erl index d8a5519195..b734d7fd98 100644 --- a/lib/kernel/test/gen_udp_SUITE.erl +++ b/lib/kernel/test/gen_udp_SUITE.erl @@ -400,6 +400,7 @@ open_fd(Config) when is_list(Config) -> {ok,S1} = gen_udp:open(0), {ok,P2} = inet:port(S1), {ok,FD} = prim_inet:getfd(S1), + {error,einval} = gen_udp:open(P2, [inet6, {fd,FD}]), {ok,S2} = gen_udp:open(P2, [{fd,FD}]), {ok,S3} = gen_udp:open(0), {ok,P3} = inet:port(S3), diff --git a/lib/odbc/test/mysql.erl b/lib/odbc/test/mysql.erl index 49068c4356..c990793213 100644 --- a/lib/odbc/test/mysql.erl +++ b/lib/odbc/test/mysql.erl @@ -26,7 +26,22 @@ %------------------------------------------------------------------------- connection_string() -> - "DSN=MySQL;Database=odbctest;Uid=odbctest;Pwd=gurka;CHARSET=utf8;SSTMT=SET NAMES 'utf8';". + case test_server:os_type() of + {unix, linux} -> + "DSN=MySQL;Database=odbctest;Uid=odbctest;Pwd=gurka;CHARSET=utf8;SSTMT=SET NAMES 'utf8';"; + {unix, sunos} -> + solaris_str(); + {unix, darwin} -> + "DSN=MySQLMac;Database=odbctest;Uid=odbctest;Pwd=gurka;CHARSET=utf8;SSTMT=SET NAMES 'utf8';" + end. + +solaris_str() -> + case erlang:system_info(system_architecture) of + "sparc" ++ _ -> + "DSN=MySQLSolaris10;Database=odbctest;Uid=odbctest;Pwd=gurka;CHARSET=utf8;SSTMT=SET NAMES 'utf8';"; + "i386" ++ _ -> + "DSN=MySQLSolaris10i386;Database=odbctest;Uid=odbctest;Pwd=gurka;CHARSET=utf8;SSTMT=SET NAMES 'utf8';" + end. %------------------------------------------------------------------------- insert_result() -> diff --git a/lib/odbc/test/odbc_connect_SUITE.erl b/lib/odbc/test/odbc_connect_SUITE.erl index e59be772e3..a076c4dfff 100644 --- a/lib/odbc/test/odbc_connect_SUITE.erl +++ b/lib/odbc/test/odbc_connect_SUITE.erl @@ -76,20 +76,26 @@ end_per_group(_GroupName, Config) -> %% Note: This function is free to add any key/value pairs to the Config %% variable, but should NOT alter/remove any existing entries. %%-------------------------------------------------------------------- -init_per_suite(Config) -> - case (catch odbc:start()) of - ok -> - case catch odbc:connect(?RDBMS:connection_string(), - [{auto_commit, off}] ++ odbc_test_lib:platform_options()) of - {ok, Ref} -> - odbc:disconnect(Ref), - [{tableName, odbc_test_lib:unique_table_name()} | Config]; - _ -> - {skip, "ODBC is not properly setup"} - end; - _ -> - {skip,"ODBC not startable"} - end. +init_per_suite(Config) when is_list(Config) -> + case odbc_test_lib:skip() of + true -> + {skip, "ODBC not supported"}; + false -> + case (catch odbc:start()) of + ok -> + case catch odbc:connect(?RDBMS:connection_string(), + [{auto_commit, off}] ++ odbc_test_lib:platform_options()) of + {ok, Ref} -> + odbc:disconnect(Ref), + [{tableName, odbc_test_lib:unique_table_name()} | Config]; + _ -> + {skip, "ODBC is not properly setup"} + end; + _ -> + {skip,"ODBC not startable"} + end + end. + %%-------------------------------------------------------------------- %% Function: end_per_suite(Config) -> _ %% Config - [tuple()] diff --git a/lib/odbc/test/odbc_data_type_SUITE.erl b/lib/odbc/test/odbc_data_type_SUITE.erl index 099ae0aa7d..323190dd99 100644 --- a/lib/odbc/test/odbc_data_type_SUITE.erl +++ b/lib/odbc/test/odbc_data_type_SUITE.erl @@ -115,14 +115,18 @@ end_per_group(_GroupName, Config) -> %% Note: This function is free to add any key/value pairs to the Config %% variable, but should NOT alter/remove any existing entries. %%-------------------------------------------------------------------- -init_per_suite(Config) -> - case (catch odbc:start()) of - ok -> - [{tableName, odbc_test_lib:unique_table_name()} | Config]; - _ -> - {skip, "ODBC not startable"} +init_per_suite(Config) when is_list(Config) -> + case odbc_test_lib:skip() of + true -> + {skip, "ODBC not supported"}; + false -> + case (catch odbc:start()) of + ok -> + [{tableName, odbc_test_lib:unique_table_name()}| Config]; + _ -> + {skip, "ODBC not startable"} + end end. - %%-------------------------------------------------------------------- %% Function: end_per_suite(Config) -> _ %% Config - [tuple()] diff --git a/lib/odbc/test/odbc_query_SUITE.erl b/lib/odbc/test/odbc_query_SUITE.erl index 76a214d553..1852678b4b 100644 --- a/lib/odbc/test/odbc_query_SUITE.erl +++ b/lib/odbc/test/odbc_query_SUITE.erl @@ -89,6 +89,7 @@ init_per_group(scrollable_cursors, Config) -> _ -> Config end; + init_per_group(_,Config) -> Config. @@ -105,11 +106,16 @@ end_per_group(_GroupName, Config) -> %% variable, but should NOT alter/remove any existing entries. %%-------------------------------------------------------------------- init_per_suite(Config) when is_list(Config) -> - case (catch odbc:start()) of - ok -> - [{tableName, odbc_test_lib:unique_table_name()}| Config]; - _ -> - {skip, "ODBC not startable"} + case odbc_test_lib:skip() of + true -> + {skip, "ODBC not supported"}; + false -> + case (catch odbc:start()) of + ok -> + [{tableName, odbc_test_lib:unique_table_name()}| Config]; + _ -> + {skip, "ODBC not startable"} + end end. %%-------------------------------------------------------------------- diff --git a/lib/odbc/test/odbc_start_SUITE.erl b/lib/odbc/test/odbc_start_SUITE.erl index 440c0ca921..e3a3440559 100644 --- a/lib/odbc/test/odbc_start_SUITE.erl +++ b/lib/odbc/test/odbc_start_SUITE.erl @@ -39,11 +39,18 @@ %% variable, but should NOT alter/remove any existing entries. %%-------------------------------------------------------------------- init_per_suite(Config) -> - case code:which(odbc) of - non_existing -> - {skip, "No ODBC built"}; - _ -> - [{tableName, odbc_test_lib:unique_table_name()} | Config] + case odbc_test_lib:skip() of + true -> + {skip, "ODBC not supported"}; + false -> + case code:which(odbc) of + non_existing -> + {skip, "No ODBC built"}; + _ -> + %% Make sure odbc is not already started + odbc:stop(), + [{tableName, odbc_test_lib:unique_table_name()} | Config] + end end. %%-------------------------------------------------------------------- diff --git a/lib/odbc/test/odbc_test.hrl b/lib/odbc/test/odbc_test.hrl index 397d04756b..f7bb338a7f 100644 --- a/lib/odbc/test/odbc_test.hrl +++ b/lib/odbc/test/odbc_test.hrl @@ -25,14 +25,16 @@ -define(RDBMS, case os:type() of {unix, sunos} -> - postgres; + mysql; {unix,linux} -> - case erlang:system_info(wordsize) of + case erlang:system_info({wordsize, external}) of 4 -> mysql; _ -> postgres end; + {unix, darwin} -> + mysql; {win32, _} -> sqlserver end). diff --git a/lib/odbc/test/odbc_test_lib.erl b/lib/odbc/test/odbc_test_lib.erl index 3e78105cf3..2d6bf5fcac 100644 --- a/lib/odbc/test/odbc_test_lib.erl +++ b/lib/odbc/test/odbc_test_lib.erl @@ -36,7 +36,7 @@ match_float(Float, Match, Delta) -> (Float < Match + Delta) and (Float > Match - Delta). odbc_check() -> - case erlang:system_info(wordsize) of + case erlang:system_info({wordsize, external}) of 4 -> ok; Other -> @@ -71,9 +71,36 @@ strict(_,_) -> ok. platform_options() -> + []. + +skip() -> case os:type() of + {unix, linux} -> + Issue = linux_issue(), + is_sles9(Issue); {unix, sunos} -> - [{scrollable_cursors, off}]; + not supported_solaris(); + _ -> + false + end. + +supported_solaris() -> + case os:version() of + {_,10,_} -> + true; _ -> - [] + false end. + +linux_issue() -> + {ok, Binary} = file:read_file("/etc/issue"), + string:tokens(binary_to_list(Binary), " "). + +is_sles11(IssueTokens) -> + lists:member(11, IssueTokens). + +is_sles10(IssueTokens) -> + lists:member(10, IssueTokens). + +is_sles9(IssueTokens) -> + lists:member(9, IssueTokens). diff --git a/lib/odbc/test/postgres.erl b/lib/odbc/test/postgres.erl index 26a2913d46..d564dbd5ff 100644 --- a/lib/odbc/test/postgres.erl +++ b/lib/odbc/test/postgres.erl @@ -30,7 +30,7 @@ connection_string() -> {unix, sunos} -> "DSN=Postgres;UID=odbctest"; {unix, linux} -> - Size = erlang:system_info(wordsize), + Size = erlang:system_info({wordsize, external}), linux_dist_connection_string(Size) end. @@ -43,7 +43,12 @@ linux_dist_connection_string(4) -> end; linux_dist_connection_string(_) -> - "DSN=PostgresLinux64;UID=odbctest". + case linux_dist() of + "ubuntu" -> + "DSN=PostgresLinux64Ubuntu;UID=odbctest"; + _ -> + "DSN=PostgresLinux64;UID=odbctest" + end. linux_dist() -> case file:read_file("/etc/issue") of diff --git a/lib/public_key/asn1/README b/lib/public_key/asn1/README index 5fb8cf9725..2a880e2d51 100644 --- a/lib/public_key/asn1/README +++ b/lib/public_key/asn1/README @@ -46,6 +46,6 @@ diff -r1.1 PKIXAttributeCertificate.asn1 --- > version AttCertVersion, -- version is v2 -4. Defenitions of publuic keys from PKCS-1.asn1 present in +4. Definitions of public keys from PKCS-1.asn1 present in PKIX1Algorithms88.asn1 where removed as we take them directly from PKCS-1.asn1
\ No newline at end of file diff --git a/lib/public_key/doc/src/public_key.xml b/lib/public_key/doc/src/public_key.xml index d60d91cd83..9a3832c68b 100644 --- a/lib/public_key/doc/src/public_key.xml +++ b/lib/public_key/doc/src/public_key.xml @@ -63,7 +63,7 @@ <p><code>pki_asn1_type() = 'Certificate' | 'RSAPrivateKey'| 'RSAPublicKey' 'DSAPrivateKey' | 'DSAPublicKey' | 'DHParameter' | 'SubjectPublicKeyInfo'</code></p> - <p><code>pem_entry () = {pki_asn1_type(), binary() %% DER or encrypted DER + <p><code>pem_entry () = {pki_asn1_type(), binary(), %% DER or encrypted DER not_encrypted | {"DES-CBC" | "DES-EDE3-CBC", crypto:rand_bytes(8)}}.</code></p> <p><code>rsa_public_key() = #'RSAPublicKey'{}</code></p> @@ -72,8 +72,6 @@ <p><code>dsa_public_key() = {integer(), #'Dss-Parms'{}} </code></p> - <p><code>rsa_private_key() = #'RSAPrivateKey'{} </code></p> - <p><code>dsa_private_key() = #'DSAPrivateKey'{}</code></p> <p><code> public_crypt_options() = [{rsa_pad, rsa_padding()}]. </code></p> @@ -149,7 +147,7 @@ <name>der_decode(Asn1type, Der) -> term()</name> <fsummary> Decodes a public key asn1 der encoded entity.</fsummary> <type> - <v>Asn1Type = atom() -</v> + <v>Asn1Type = atom()</v> <d> ASN.1 type present in the public_key applications asn1 specifications.</d> <v>Der = der_encoded()</v> @@ -166,7 +164,8 @@ <v>Asn1Type = atom()</v> <d> Asn1 type present in the public_key applications ASN.1 specifications.</d> - <v>Entity = term() - The erlang representation of <c> Asn1Type</c></v> + <v>Entity = term()</v> + <d>The erlang representation of <c>Asn1Type</c></d> </type> <desc> <p> Encodes a public key entity with ASN.1 DER encoding.</p> @@ -218,12 +217,13 @@ <fsummary> Creates a pem entry that can be fed to pem_encode/1.</fsummary> <type> <v>Asn1Type = pki_asn1_type()</v> - <v>Entity = term() - The Erlang representation of + <v>Entity = term()</v> + <d>The Erlang representation of <c>Asn1Type</c>. If <c>Asn1Type</c> is 'SubjectPublicKeyInfo' then <c>Entity</c> must be either an rsa_public_key() or a dsa_public_key() and this function will create the appropriate 'SubjectPublicKeyInfo' entry. - </v> + </d> <v>CipherInfo = {"DES-CBC" | "DES-EDE3-CBC", crypto:rand_bytes(8)}</v> <v>Password = string()</v> </type> @@ -281,7 +281,7 @@ <desc> <p>Der encodes a pkix x509 certificate or part of such a certificate. This function must be used for encoding certificates or parts of certificates - that are decoded/created on the otp format, whereas for the plain format this + that are decoded/created in the otp format, whereas for the plain format this function will directly call der_encode/2. </p> </desc> </func> diff --git a/lib/public_key/src/public_key.erl b/lib/public_key/src/public_key.erl index 2901020e83..33fcce2c44 100644 --- a/lib/public_key/src/public_key.erl +++ b/lib/public_key/src/public_key.erl @@ -488,9 +488,10 @@ pkix_path_validation(PathErr, [Cert | Chain], Options0) when is_atom(PathErr)-> _:_ -> {error, Reason} end; -pkix_path_validation(TrustedCert, CertChain, Options) when - is_binary(TrustedCert) -> OtpCert = pkix_decode_cert(TrustedCert, - otp), pkix_path_validation(OtpCert, CertChain, Options); +pkix_path_validation(TrustedCert, CertChain, Options) + when is_binary(TrustedCert) -> + OtpCert = pkix_decode_cert(TrustedCert, otp), + pkix_path_validation(OtpCert, CertChain, Options); pkix_path_validation(#'OTPCertificate'{} = TrustedCert, CertChain, Options) when is_list(CertChain), is_list(Options) -> diff --git a/lib/sasl/src/release_handler_1.erl b/lib/sasl/src/release_handler_1.erl index 3d6bc9fbc3..ff62f847ac 100644 --- a/lib/sasl/src/release_handler_1.erl +++ b/lib/sasl/src/release_handler_1.erl @@ -33,7 +33,7 @@ %% libdirs = [{Lib, LibVsn, LibDir}] - Maps Lib to Vsn and Directory %% unpurged = [{Mod, soft_purge | brutal_purge}] %% vsns = [{Mod, OldVsn, NewVsn}] - remember the old vsn of a mod -%% before it is removed/a new vsn is loaded; the new vsn +%% before a new vsn is loaded; the new vsn %% is kept in case of a downgrade, where the code_change %% function receives the vsn of the module to downgrade %% *to*. @@ -224,7 +224,7 @@ eval({load_object_code, {Lib, LibVsn, Modules}}, EvalState) -> FName = filename:join([LibDir, "ebin", File]), case erl_prim_loader:get_file(FName) of {ok, Bin, FName2} -> - NVsns = add_new_vsn(Mod, Bin, Vsns), + NVsns = add_vsns(Mod, Bin, Vsns), {[{Mod, Bin, FName2} | Bins],NVsns}; error -> throw({error, {no_such_file,FName}}) @@ -258,32 +258,21 @@ eval({load, {Mod, _PrePurgeMethod, PostPurgeMethod}}, EvalState) -> {value, {_Mod, Bin, File}} = lists:keysearch(Mod, 1, Bins), % load_binary kills all procs running old code % if soft_purge, we know that there are no such procs now - Vsns = EvalState#eval_state.vsns, - NewVsns = add_old_vsn(Mod, Vsns), code:load_binary(Mod, File, Bin), % Now, the prev current is old. There might be procs % running it. Find them. Unpurged = do_soft_purge(Mod,PostPurgeMethod,EvalState#eval_state.unpurged), EvalState#eval_state{bins = lists:keydelete(Mod, 1, Bins), - unpurged = Unpurged, - vsns = NewVsns}; + unpurged = Unpurged}; eval({remove, {Mod, _PrePurgeMethod, PostPurgeMethod}}, EvalState) -> - % purge kills all procs running old code - % if soft_purge, we know that there are no such procs now - Vsns = EvalState#eval_state.vsns, - NewVsns = add_old_vsn(Mod, Vsns), + %% purge kills all procs running old code + %% if soft_purge, we know that there are no such procs now code:purge(Mod), code:delete(Mod), - % Now, the prev current is old. There might be procs - % running it. Find them. - Unpurged = - case code:soft_purge(Mod) of - true -> EvalState#eval_state.unpurged; - false -> [{Mod, PostPurgeMethod} | EvalState#eval_state.unpurged] - end, -%% Bins = EvalState#eval_state.bins, -%% EvalState#eval_state{bins = lists:keydelete(Mod, 1, Bins), - EvalState#eval_state{unpurged = Unpurged, vsns = NewVsns}; + %% Now, the prev current is old. There might be procs + %% running it. Find them. + Unpurged = do_soft_purge(Mod,PostPurgeMethod,EvalState#eval_state.unpurged), + EvalState#eval_state{unpurged = Unpurged}; eval({purge, Modules}, EvalState) -> % Now, if there are any processes still executing old code, OR % if some new processes started after suspend but before load, @@ -606,26 +595,20 @@ sync_nodes(Id, Nodes) -> end, NNodes). -add_old_vsn(Mod, Vsns) -> +add_vsns(Mod, NewBin, Vsns) -> + OldVsn = get_current_vsn(Mod), + NewVsn = get_vsn(NewBin), case lists:keysearch(Mod, 1, Vsns) of - {value, {Mod, undefined, NewVsn}} -> - OldVsn = get_current_vsn(Mod), - lists:keyreplace(Mod, 1, Vsns, {Mod, OldVsn, NewVsn}); - {value, {Mod, _OldVsn, _NewVsn}} -> - Vsns; + {value, {Mod, OldVsn0, NewVsn0}} -> + lists:keyreplace(Mod, 1, Vsns, {Mod, + replace_undefined(OldVsn0,OldVsn), + replace_undefined(NewVsn0,NewVsn)}); false -> - OldVsn = get_current_vsn(Mod), - [{Mod, OldVsn, undefined} | Vsns] + [{Mod, OldVsn, NewVsn} | Vsns] end. -add_new_vsn(Mod, Bin, Vsns) -> - NewVsn = get_vsn(Bin), - case lists:keysearch(Mod, 1, Vsns) of - {value, {Mod, OldVsn, undefined}} -> - lists:keyreplace(Mod, 1, Vsns, {Mod, OldVsn, NewVsn}); - false -> - [{Mod, undefined, NewVsn} | Vsns] - end. +replace_undefined(undefined,Vsn) -> Vsn; +replace_undefined(Vsn,_) -> Vsn. %%----------------------------------------------------------------- %% Func: get_current_vsn/1 @@ -645,7 +628,9 @@ get_current_vsn(Mod) -> {ok, Bin, _File2} -> get_vsn(Bin); error -> - throw({error, {no_such_file, File}}) + %% This is the case when a new module is added, there will + %% be no current version of it at the time of this call. + undefined end. %%----------------------------------------------------------------- diff --git a/lib/sasl/test/release_handler_SUITE.erl b/lib/sasl/test/release_handler_SUITE.erl index d1939694ea..9c7733b7ec 100644 --- a/lib/sasl/test/release_handler_SUITE.erl +++ b/lib/sasl/test/release_handler_SUITE.erl @@ -55,7 +55,7 @@ win32_cases() -> %% Cases that can be run on all platforms cases() -> - [otp_2740, otp_2760, otp_5761, otp_9402, instructions, eval_appup]. + [otp_2740, otp_2760, otp_5761, otp_9402, otp_9417, instructions, eval_appup]. groups() -> [{release,[], @@ -683,7 +683,7 @@ otp_9402(Conf) when is_list(Conf) -> %% Check path Dir1 = filename:join([LibDir, "a-1.1"]), Dir1 = rpc:call(Node, code, lib_dir, [a]), - ABeam = code:which(a), + ABeam = rpc:call(Node, code, which, [a]), %% Install second release, with no changed modules {ok, RelVsn2} = @@ -696,7 +696,6 @@ otp_9402(Conf) when is_list(Conf) -> ok = rpc:call(Node, release_handler, install_file, [RelVsn2, filename:join(Rel2Dir, "sys.config")]), - %% Install RelVsn2 {ok, RelVsn1, []} = rpc:call(Node, release_handler, install_release, [RelVsn2]), @@ -707,7 +706,7 @@ otp_9402(Conf) when is_list(Conf) -> true = filelib:is_regular(filename:join(APrivDir2,"file")), %% Just to make sure no modules have been re-loaded - ABeam = code:which(a), + ABeam = rpc:call(Node, code, which, [a]), %% Install RelVsn1 again {ok, _OtherVsn, []} = @@ -719,11 +718,65 @@ otp_9402(Conf) when is_list(Conf) -> false = filelib:is_regular(filename:join(APrivDir1,"file")), %% Just to make sure no modules have been re-loaded - ABeam = code:which(a), + ABeam = rpc:call(Node, code, which, [a]), ok. +%% When a module is deleted in an appup instruction, the upgrade +%% failed if the module was not loaded. +otp_9417(Conf) when is_list(Conf) -> + %% Set some paths + PrivDir = priv_dir(Conf), + Dir = filename:join(PrivDir,"otp_9417"), + LibDir = filename:join(?config(data_dir, Conf), "lib"), + + %% Create the releases + Rel1 = create_and_install_fake_first_release(Dir, + [{b,"1.0",LibDir}]), + Rel2 = create_fake_upgrade_release(Dir, + "2", + [{b,"2.0",LibDir}], + {Rel1,Rel1,[LibDir]}), + Rel1Dir = filename:dirname(Rel1), + Rel2Dir = filename:dirname(Rel2), + + %% Start a slave node + {ok, Node} = t_start_node(otp_9417, Rel1, filename:join(Rel1Dir,"sys.config")), + + %% Check paths + Dir1 = filename:join([LibDir, "b-1.0"]), + Dir1 = rpc:call(Node, code, lib_dir, [b]), + BLibBeam = filename:join([Dir1,"ebin","b_lib.beam"]), + BLibBeam = rpc:call(Node,code,which,[b_lib]), + false = rpc:call(Node,code,is_loaded,[b_lib]), + false = rpc:call(Node,code,is_loaded,[b_server]), + + %% Install second release, which removes b_lib module + {ok, RelVsn2} = + rpc:call(Node, release_handler, set_unpacked, + [Rel2++".rel", [{b,"2.0",LibDir}]]), + ok = rpc:call(Node, release_handler, install_file, + [RelVsn2, filename:join(Rel2Dir, "relup")]), + ok = rpc:call(Node, release_handler, install_file, + [RelVsn2, filename:join(Rel2Dir, "start.boot")]), + ok = rpc:call(Node, release_handler, install_file, + [RelVsn2, filename:join(Rel2Dir, "sys.config")]), + + {ok, _RelVsn1, []} = + rpc:call(Node, release_handler, install_release, [RelVsn2]), + + %% Check that the module does no longer exist + false = rpc:call(Node, code, is_loaded, [b_lib]), + non_existing = rpc:call(Node, code, which, [b_lib]), + + %% And check some paths + Dir2 = filename:join([LibDir, "b-2.0"]), + Dir2 = rpc:call(Node, code, lib_dir, [b]), + BServerBeam = filename:join([Dir2,"ebin","b_server.beam"]), + {file,BServerBeam} = rpc:call(Node,code,is_loaded,[b_server]), + ok. + %% Test upgrade and downgrade of applications eval_appup(Conf) when is_list(Conf) -> diff --git a/lib/sasl/test/release_handler_SUITE_data/Makefile.src b/lib/sasl/test/release_handler_SUITE_data/Makefile.src index abaa08f810..a12e526d2e 100644 --- a/lib/sasl/test/release_handler_SUITE_data/Makefile.src +++ b/lib/sasl/test/release_handler_SUITE_data/Makefile.src @@ -1,16 +1,19 @@ EFLAGS=+debug_info P2B= \ - P2B/a-2.0/ebin/a.beam \ - P2B/a-2.0/ebin/a_sup.beam + P2B/a-2.0/ebin/a.@EMULATOR@ \ + P2B/a-2.0/ebin/a_sup.@EMULATOR@ LIB= \ - lib/a-1.2/ebin/a.beam \ - lib/a-1.2/ebin/a_sup.beam \ - lib/a-1.1/ebin/a.beam \ - lib/a-1.1/ebin/a_sup.beam \ - lib/a-1.0/ebin/a.beam \ - lib/a-1.0/ebin/a_sup.beam \ + lib/a-1.2/ebin/a.@EMULATOR@ \ + lib/a-1.2/ebin/a_sup.@EMULATOR@ \ + lib/a-1.1/ebin/a.@EMULATOR@ \ + lib/a-1.1/ebin/a_sup.@EMULATOR@ \ + lib/a-1.0/ebin/a.@EMULATOR@ \ + lib/a-1.0/ebin/a_sup.@EMULATOR@ \ + lib/b-1.0/ebin/b_server.@EMULATOR@ \ + lib/b-1.0/ebin/b_lib.@EMULATOR@ \ + lib/b-2.0/ebin/b_server.@EMULATOR@ APP= \ app1_app2/lib1/app1-1.0/ebin/app1_sup.@EMULATOR@ \ @@ -64,6 +67,15 @@ lib/a-1.2/ebin/a.@EMULATOR@: lib/a-1.2/src/a.erl lib/a-1.2/ebin/a_sup.@EMULATOR@: lib/a-1.2/src/a_sup.erl erlc $(EFLAGS) -olib/a-1.2/ebin lib/a-1.2/src/a_sup.erl +lib/b-1.0/ebin/b_server.@EMULATOR@: lib/b-1.0/src/b_server.erl + erlc $(EFLAGS) -olib/b-1.0/ebin lib/b-1.0/src/b_server.erl +lib/b-1.0/ebin/b_lib.@EMULATOR@: lib/b-1.0/src/b_lib.erl + erlc $(EFLAGS) -olib/b-1.0/ebin lib/b-1.0/src/b_lib.erl + +lib/b-2.0/ebin/b_server.@EMULATOR@: lib/b-2.0/src/b_server.erl + erlc $(EFLAGS) -olib/b-2.0/ebin lib/b-2.0/src/b_server.erl + + app1_app2/lib1/app1-1.0/ebin/app1_sup.@EMULATOR@: app1_app2/lib1/app1-1.0/src/app1_sup.erl erlc $(EFLAGS) -oapp1_app2/lib1/app1-1.0/ebin app1_app2/lib1/app1-1.0/src/app1_sup.erl diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/README b/lib/sasl/test/release_handler_SUITE_data/lib/README index e539580ac0..667d21d4cf 100644 --- a/lib/sasl/test/release_handler_SUITE_data/lib/README +++ b/lib/sasl/test/release_handler_SUITE_data/lib/README @@ -6,4 +6,11 @@ can be upgraded to from a-1.0. Module a has changed a-1.2: can be upgraded to from a-1.1. -No module have changed, but priv dir is added including one 'file'
\ No newline at end of file +No module have changed, but priv dir is added including one 'file' + +b-1.0: +start version, includes b_lib and b_server + +b-2.0: +can be upgraded to from b-1.0. +Removes b_lib and updates b_server diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/ebin/b.app b/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/ebin/b.app new file mode 100644 index 0000000000..00347b2754 --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/ebin/b.app @@ -0,0 +1,7 @@ +%% -*- erlang -*- +{application, b, + [{description, "B CXC 138 12"}, + {vsn, "1.0"}, + {modules, [{b_server, 1},{b_lib, 1}]}, + {registered, [b_server]}, + {applications, [kernel, stdlib]}]}. diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/src/b_lib.erl b/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/src/b_lib.erl new file mode 100644 index 0000000000..7e8a308a5e --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/src/b_lib.erl @@ -0,0 +1,3 @@ +-module(b_lib). +-compile(export_all). +foo() -> ok. diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/src/b_server.erl b/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/src/b_server.erl new file mode 100644 index 0000000000..e1a80a076f --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-1.0/src/b_server.erl @@ -0,0 +1,37 @@ +-module(b_server). + +-behaviour(gen_server). + +%% API +-export([start_link/0]). + +%% gen_server callbacks +-export([init/1, handle_call/3, handle_cast/2, handle_info/2, + terminate/2, code_change/3]). + +-define(SERVER, ?MODULE). + +-record(state, {}). + +start_link() -> + gen_server:start_link({local, ?SERVER}, ?MODULE, [], []). + +init([]) -> + {ok, #state{}}. + +handle_call(_Request, _From, State) -> + Reply = ok, + {reply, Reply, State}. + +handle_cast(_Msg, State) -> + {noreply, State}. + +handle_info(_Info, State) -> + {noreply, State}. + +terminate(_Reason, _State) -> + ok. + +code_change(OldVsn, State, Extra) -> + file:write_file("/tmp/b_server",io_lib:format("~p~n",[{"1.0",OldVsn,Extra}])), + {ok, State}. diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/ebin/b.app b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/ebin/b.app new file mode 100644 index 0000000000..73c8e42b32 --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/ebin/b.app @@ -0,0 +1,7 @@ +%% -*- erlang -*- +{application, b, + [{description, "B CXC 138 12"}, + {vsn, "2.0"}, + {modules, [{b_server, 1}]}, + {registered, [b_server]}, + {applications, [kernel, stdlib]}]}. diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/ebin/b.appup b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/ebin/b.appup new file mode 100644 index 0000000000..d261a37732 --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/ebin/b.appup @@ -0,0 +1,3 @@ +{"2.0", + [{"1.0",[{delete_module,b_lib}, {update,b_server,{advanced,[]}}]}], + [{"1.0",[{add_module,b_lib},{update,b_server,{advanced,[]}}]}]}. diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/src/b_lib.erl b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/src/b_lib.erl new file mode 100644 index 0000000000..7e8a308a5e --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/src/b_lib.erl @@ -0,0 +1,3 @@ +-module(b_lib). +-compile(export_all). +foo() -> ok. diff --git a/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/src/b_server.erl b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/src/b_server.erl new file mode 100644 index 0000000000..f8bfbdaff7 --- /dev/null +++ b/lib/sasl/test/release_handler_SUITE_data/lib/b-2.0/src/b_server.erl @@ -0,0 +1,37 @@ +-module(b_server). + +-behaviour(gen_server). + +%% API +-export([start_link/0]). + +%% gen_server callbacks +-export([init/1, handle_call/3, handle_cast/2, handle_info/2, + terminate/2, code_change/3]). + +-define(SERVER, ?MODULE). + +-record(state, {}). + +start_link() -> + gen_server:start_link({local, ?SERVER}, ?MODULE, [], []). + +init([]) -> + {ok, #state{}}. + +handle_call(_Request, _From, State) -> + Reply = ok, + {reply, Reply, State}. + +handle_cast(_Msg, State) -> + {noreply, State}. + +handle_info(_Info, State) -> + {noreply, State}. + +terminate(_Reason, _State) -> + ok. + +code_change(OldVsn, State, Extra) -> + file:write_file("/tmp/b_server",io_lib:format("~p~n",[{"2.0",OldVsn,Extra}])), + {ok, State}. diff --git a/lib/snmp/Makefile b/lib/snmp/Makefile index 20e3d4692a..c55eff04c6 100644 --- a/lib/snmp/Makefile +++ b/lib/snmp/Makefile @@ -67,7 +67,7 @@ do_configure: configure configure: configure.in autoconf -.PHONY: info +.PHONY: info gclean info: @echo "OS: $(OS)" @@ -76,6 +76,9 @@ info: @echo "SNMP_VSN: $(SNMP_VSN)" @echo "APP_VSN: $(APP_VSN)" +gclean: + git clean -fXd + # ---------------------------------------------------- # Application (source) release targets diff --git a/lib/snmp/doc/src/Makefile b/lib/snmp/doc/src/Makefile index 35ed63e103..aa9431477c 100644 --- a/lib/snmp/doc/src/Makefile +++ b/lib/snmp/doc/src/Makefile @@ -152,6 +152,7 @@ $(TOP_PDF_FILE): $(XML_FILES) pdf: $(TOP_PDF_FILE) html: gifs $(HTML_REF_MAN_FILE) +html2: html $(INDEX_TARGET) clean clean_docs: clean_html clean_man clean_pdf rm -f errs core *~ @@ -228,6 +229,7 @@ clean_man: clean_html: @echo "cleaning html:" rm -rf $(HTMLDIR)/* + rm -f $(INDEX_TARGET) $(MAN7DIR)/%.7: $(MIBSDIR)/%.mib @echo "processing $*" @@ -286,7 +288,7 @@ release_docs_spec: docs $(INSTALL_DATA) $(INFO_FILE) $(RELSYSDIR) $(INSTALL_DIR) $(RELEASE_PATH)/man/man1 $(INSTALL_DATA) $(MAN1_FILES) $(RELEASE_PATH)/man/man1 - $(INSTALL_DIR) $(RELEASE_PATH)/man/man + $(INSTALL_DIR) $(RELEASE_PATH)/man/man3 $(INSTALL_DATA) $(MAN3_FILES) $(RELEASE_PATH)/man/man3 $(INSTALL_DIR) $(RELEASE_PATH)/man/man6 $(INSTALL_DATA) $(MAN6_FILES) $(RELEASE_PATH)/man/man6 diff --git a/lib/snmp/doc/src/notes.xml b/lib/snmp/doc/src/notes.xml index 6a20d8ee3a..4178192120 100644 --- a/lib/snmp/doc/src/notes.xml +++ b/lib/snmp/doc/src/notes.xml @@ -33,6 +33,136 @@ </header> <section> + <title>SNMP Development Toolkit 4.21</title> + <p>Version 4.21 supports code replacement in runtime from/to + version 4.20.1, 4.20 and 4.19. </p> + + <section> + <title>Improvements and new features</title> +<!-- + <p>-</p> +--> + <list type="bulleted"> + <item> + <p>[manager] There was no way to specify transport domain. + The transport domains was assumed to be IPv4 (transportDomainUdpIpv4). + This has now been changed so that it can also be IPv6 + (transportDomainUdpIpv6). + To facilitate this, the transport domain, <c>tdomain</c>, + is now a (new) valid option when + <seealso marker="snmpm#register_agent">registering</seealso> + a new agent (and + <seealso marker="snmpm#update_agent_info">updating</seealso> + agent info). </p> + <p>This also mean that the transport behaviour has changed. </p> + <p>Own Id: OTP-9305</p> + <p>Aux Id: Seq 11847</p> + </item> + + <item> + <p>[compiler] Added the option + <seealso marker="snmpc#compile">warnings_as_errors</seealso> + (for the SNMP MIB compiler (escript) frontend, the option + <seealso marker="snmpc(command)#option_wae">--wae</seealso> is used) + which specifies whether warnings should be treated as errors. </p> + <p>Tuncer Ayaz</p> + <p>Own Id: OTP-9437</p> + </item> + </list> + + </section> + + <section> + <title>Fixed Bugs and Malfunctions</title> +<!-- + <p>-</p> +--> + + <list type="bulleted"> + <item> + <p>The snmp config tool could not handle (manager) audit trail config + because the option seqno was not handled. </p> + <p>Own Id: OTP-9354</p> + </item> + + <item> + <p>[agent] The SNMP ACM cache was not properly updated when + changes where made to the VACM security-to-group, access and + view-tree-family tables. </p> + <p>Own Id: OTP-9367</p> + <p>Aux Id: Seq 11858</p> + </item> + + <item> + <p>Fixed install directory typo for man3. </p> + <p>Peter Lemenkov</p> + <p>Hans Ulrich Niedermann</p> + <p>Own Id: OTP-9442</p> + </item> + + </list> + </section> + + + <section> + <title>Incompatibilities</title> + <p>-</p> + </section> + + </section> <!-- 4.21 --> + + + <section> + <title>SNMP Development Toolkit 4.20.1</title> + <p>Version 4.20.1 supports code replacement in runtime from/to + version 4.20, 4.19 and 4.18.</p> + + <section> + <title>Improvements and new features</title> + <p>-</p> +<!-- + <list type="bulleted"> + <item> + <p>Added type specs for functions that do not return. </p> + <p>Kostis Sagonas</p> + <p>Own Id: OTP-9208</p> + </item> + </list> +--> + </section> + + <section> + <title>Fixed Bugs and Malfunctions</title> +<!-- + <p>-</p> +--> + <list type="bulleted"> + <item> + <p>[agent] Did not handle transport domains properly in some cases, + for instance trap sending. </p> + <p>Own Id: OTP-9400</p> + </item> + + <item> + <p>[agent] Wrong default transport domain, snmpUDPDomain, instead + of transportDomainUdpIpv4. </p> + <p>Own Id: OTP-9425</p> + <p>Aux Id: Seq 11874</p> + </item> + + </list> + </section> + + + <section> + <title>Incompatibilities</title> + <p>-</p> + </section> + + </section> <!-- 4.20.1 --> + + + <section> <title>SNMP Development Toolkit 4.20</title> <p>Version 4.20 supports code replacement in runtime from/to version 4.19 and 4.18.</p> diff --git a/lib/snmp/doc/src/snmpc.xml b/lib/snmp/doc/src/snmpc.xml index 771629492d..61d19251c5 100644 --- a/lib/snmp/doc/src/snmpc.xml +++ b/lib/snmp/doc/src/snmpc.xml @@ -48,7 +48,11 @@ <type> <v>File = string()</v> <v>Options = [opt()]</v> - <v>opt() = db() | relaxed_row_name_assign_check() | deprecated() | description() | reference() | group_check() | i() | il() | imports() | module() | module_identity() | module_compliance() | agent_capabilities() | outdir() | no_defs() | verbosity() | warnings()</v> + <v>opt() = db() | relaxed_row_name_assign_check() | deprecated() | + description() | reference() | group_check() | i() | il() | + imports() | module() | module_identity() | module_compliance() | + agent_capabilities() | outdir() | no_defs() | verbosity() | + warnings() | warnings_as_errors()</v> <v>db() = {db, volatile|persistent|mnesia}</v> <v>deprecated() = {deprecated, bool()}</v> <v>relaxed_row_name_assign_check() = relaxed_row_name_assign_check</v> @@ -66,6 +70,7 @@ <v>outdir() = {outdir, dir()}</v> <v>verbosity() = {verbosity, silence|warning|info|log|debug|trace}</v> <v>warnings() = {warnings, bool()}</v> + <v>warnings_as_errors() = warnings_as_errors</v> <v>dir() = string()</v> <v>BinFileName = string()</v> </type> @@ -200,11 +205,17 @@ <item> <p>The option <c>warnings</c> specifies whether warning - messages should be shown. </p> + messages should be shown. </p> <p>Default is <c>true</c>. </p> </item> + <item> + <p>The option <c>warnings_as_errors</c>, if present, specifies + whether warnings should be treated as errors.</p> + </item> + </list> + <p>The MIB compiler understands both SMIv1 and SMIv2 MIBs. It uses the <c>MODULE-IDENTITY</c> statement to determine if the MIB is version 1 or 2. diff --git a/lib/snmp/doc/src/snmpc_cmd.xml b/lib/snmp/doc/src/snmpc_cmd.xml index 9358382a10..72116f8981 100644 --- a/lib/snmp/doc/src/snmpc_cmd.xml +++ b/lib/snmp/doc/src/snmpc_cmd.xml @@ -50,6 +50,8 @@ with definitions of Erlang constants for the objects in the MIB, see <seealso marker="snmpc#mib_to_hrl">mib_to_hrl/1</seealso>. </p> + + <marker id="options"></marker> </desc> </func> </funcs> @@ -58,15 +60,18 @@ <title>Compiler options</title> <p>The following options are supported (note that most of these relate to the compilation of the MIB file):</p> + <marker id="option_help"></marker> <taglist> <tag>--help</tag> <item> <p>Prints help info.</p> + <marker id="option_version"></marker> </item> <tag>--version</tag> <item> <p>Prints application and mib format version.</p> + <marker id="option_verbosity"></marker> </item> <tag>--verbosity <em>verbosity</em></tag> @@ -74,11 +79,20 @@ <p>Print debug info. </p> <p><c>verbosity</c> = <c>trace</c> | <c>debug</c> | <c>log</c> | <c>info</c> | <c>silence</c></p> <p>Defaults to <c>silence</c>.</p> + <marker id="option_warnings"></marker> </item> <tag>--warnings</tag> <item> <p>Print warning messages. </p> + <marker id="option_wae"></marker> + </item> + + <tag>--wae</tag> + <item> + <p>Warnings as errors. + Indicates that warnings shall be treated as errors. </p> + <marker id="option_odir"></marker> </item> <tag>--o <em>directory</em></tag> @@ -86,6 +100,7 @@ <p>The directory where the compiler should place the output files. If not specified, output files will be placed in the current working directory.</p> + <marker id="option_idir"></marker> </item> <tag>--i <em>Directory</em></tag> @@ -94,6 +109,7 @@ By default, the current working directory is always included. </p> <p>This option can be present several times, each time specifying <em>one</em> path. </p> + <marker id="option_ildir"></marker> </item> <tag>--il <em>Directory</em></tag> @@ -106,6 +122,7 @@ the current version may be in the system). The current directory and the "snmp-home"/priv/mibs/ are always listed last in the include path. </p> + <marker id="option_sgc"></marker> </item> <tag>--sgc</tag> @@ -114,42 +131,50 @@ group check of the mib compiler. That is, should the OBJECT-GROUP and the NOTIFICATION-GROUP macro(s) be checked for correctness or not. </p> + <marker id="option_dep"></marker> </item> <tag>--dep</tag> <item> <p>Keep deprecated definition(s). If not specified the compiler will ignore deprecated definitions. </p> + <marker id="option_desc"></marker> </item> <tag>--desc</tag> <item> <p>The DESCRIPTION field will be included. </p> + <marker id="option_ref"></marker> </item> <tag>--ref</tag> <item> <p>The REFERENCE field will be included. </p> + <marker id="option_imp"></marker> </item> <tag>--imp</tag> <item> <p>The IMPORTS field will be included. </p> + <marker id="option_mi"></marker> </item> <tag>--mi</tag> <item> <p>The MODULE-IDENTITY field will be included. </p> + <marker id="option_mc"></marker> </item> <tag>--mc</tag> <item> <p>The MODULE-COMPLIANCE field will be included. </p> + <marker id="option_ac"></marker> </item> <tag>--ac</tag> <item> <p>The AGENT-CAPABILITIES field will be included. </p> + <marker id="option_mod"></marker> </item> <tag>--mod <em>module</em></tag> @@ -157,6 +182,7 @@ <p>The module which implements all the instrumentation functions. </p> <p>The name of all instrumentation functions must be the same as the corresponding managed object it implements. </p> + <marker id="option_nd"></marker> </item> <tag>--nd</tag> @@ -165,6 +191,7 @@ used if a managed object have no instrumentation function. Instead this will be reported as an error, and the compilation aborts. </p> + <marker id="option_rrnac"></marker> </item> <tag>--rrnac</tag> @@ -176,6 +203,7 @@ This means that the error will be converted to a warning. </p> <p>By default it is not included, but if this option is present it will be. </p> + <marker id="see_also"></marker> </item> </taglist> diff --git a/lib/snmp/doc/src/snmpm.xml b/lib/snmp/doc/src/snmpm.xml index b527d171b0..c36a1b2a24 100644 --- a/lib/snmp/doc/src/snmpm.xml +++ b/lib/snmp/doc/src/snmpm.xml @@ -283,27 +283,27 @@ sec_level = noAuthNoPriv | authNoPriv | authPriv <v>TargetName = target_name()</v> <v>Config = [agent_config()]</v> <v>agent_config() = {Item, Val}</v> - <v>Item = engine_id | address | port | community | timeout | max_message_size | version | sec_model | sec_name | sec_level</v> + <v>Item = engine_id | address | port | community | timeout | max_message_size | version | sec_model | sec_name | sec_level | tdomain</v> <v>Val = term()</v> <v>Reason = term()</v> </type> <desc> <p>Explicitly instruct the manager to handle this agent, with - <c>UserId</c> as the responsible user. </p> - <p>Called to instruct the manager that this agent - shall be handled. This function is used when - the user knows in advance which agents the - manager shall handle. - Note that there is an alternate way to do the same thing: - Add the agent to the manager config files (see - <seealso marker="snmp_manager_config_files#agents">agents.conf</seealso>).</p> + <c>UserId</c> as the responsible user. </p> + <p>Called to instruct the manager that this agent shall be handled. + This function is used when the user knows in advance which agents + the manager shall handle. + Note that there is an alternate way to do the same thing: + Add the agent to the manager config files (see + <seealso marker="snmp_manager_config_files#agents">agents.conf</seealso>).</p> <p><c>TargetName</c> is a non-empty string, - uniquely identifying the agent. </p> - <p>The type of <c>Val</c> depends on <c>Item</c>: </p> + uniquely identifying the agent. </p> + <p>The type of <c>Val</c> depends on <c>Item</c>: </p> <code type="none"><![CDATA[ [mandatory] engine_id = string() [mandatory] address = ip_address() [optional] port = integer() +[optional] tdomain = transportDomainUdpIpv4 | transportDomainUdpIpv6 [optional] community = string() [optional] timeout = integer() | snmp_timer() [optional] max_message_size = integer() @@ -312,7 +312,9 @@ sec_level = noAuthNoPriv | authNoPriv | authPriv [optional] sec_name = string() [optional] sec_level = noAuthNoPriv | authNoPriv | authPriv ]]></code> - <p>Note that if no <c>Port</c> is given, the default value is used.</p> + <p>Note that if no <c>tdomain</c> is given, the default value, + <c>transportDomainUdpIpv4</c>, is used.</p> + <p>Note that if no <c>port</c> is given, the default value is used.</p> <marker id="unregister_agent"></marker> </desc> @@ -348,17 +350,25 @@ sec_level = noAuthNoPriv | authNoPriv | authPriv </func> <func> + <name>update_agent_info(UserId, TargetName, Info) -> ok | {error, Reason}</name> <name>update_agent_info(UserId, TargetName, Item, Val) -> ok | {error, Reason}</name> <fsummary>Update agent config</fsummary> <type> <v>UserId = term()</v> <v>TargetName = target_name()</v> - <v>Item = atom()</v> - <v>Val = term()</v> + <v>Info = [{item(), item_value()}]</v> + <v>Item = item()</v> + <v>item() = atom()</v> + <v>Val = item_value()</v> + <v>item_value() = term()</v> <v>Reason = term()</v> </type> <desc> - <p>Update agent config.</p> + <p>Update agent config. The function <c>update_agent_info/3</c> + should be used when several values needs to be updated atomically. </p> + <p>See function + <seealso marker="#register_agent">register_agent</seealso>) + for more info about what kind of items are allowed. </p> <marker id="which_agents"></marker> </desc> diff --git a/lib/snmp/src/agent/snmp_view_based_acm_mib.erl b/lib/snmp/src/agent/snmp_view_based_acm_mib.erl index 28469a7b4e..37f6dd3f26 100644 --- a/lib/snmp/src/agent/snmp_view_based_acm_mib.erl +++ b/lib/snmp/src/agent/snmp_view_based_acm_mib.erl @@ -247,6 +247,7 @@ add_sec2group(SecModel, SecName, GroupName) -> Key = [Key1, length(Key2) | Key2], case table_cre_row(vacmSecurityToGroupTable, Key, Row) of true -> + snmpa_agent:invalidate_ca_cache(), {ok, Key}; false -> {error, create_failed} @@ -260,6 +261,7 @@ add_sec2group(SecModel, SecName, GroupName) -> delete_sec2group(Key) -> case table_del_row(vacmSecurityToGroupTable, Key) of true -> + snmpa_agent:invalidate_ca_cache(), ok; false -> {error, delete_failed} @@ -279,6 +281,7 @@ add_access(GroupName, Prefix, SecModel, SecLevel, Match, RV, WV, NV) -> Key3 = [SM, SL], Key = Key1 ++ Key2 ++ Key3, snmpa_vacm:insert([{Key, Row}], false), + snmpa_agent:invalidate_ca_cache(), {ok, Key}; {error, Reason} -> {error, Reason}; @@ -287,6 +290,7 @@ add_access(GroupName, Prefix, SecModel, SecLevel, Match, RV, WV, NV) -> end. delete_access(Key) -> + snmpa_agent:invalidate_ca_cache(), snmpa_vacm:delete(Key). @@ -299,6 +303,7 @@ add_view_tree_fam(ViewIndex, SubTree, Status, Mask) -> Key = [length(Key1) | Key1] ++ [length(Key2) | Key2], case table_cre_row(vacmViewTreeFamilyTable, Key, Row) of true -> + snmpa_agent:invalidate_ca_cache(), {ok, Key}; false -> {error, create_failed} @@ -312,6 +317,7 @@ add_view_tree_fam(ViewIndex, SubTree, Status, Mask) -> delete_view_tree_fam(Key) -> case table_del_row(vacmViewTreeFamilyTable, Key) of true -> + snmpa_agent:invalidate_ca_cache(), ok; false -> {error, delete_failed} diff --git a/lib/snmp/src/agent/snmpa_agent.erl b/lib/snmp/src/agent/snmpa_agent.erl index 82a7ec647b..6322f0f21d 100644 --- a/lib/snmp/src/agent/snmpa_agent.erl +++ b/lib/snmp/src/agent/snmpa_agent.erl @@ -1626,7 +1626,7 @@ invalidate_ca_cache() -> MasterAgent ! invalidate_ca_cache; false -> %% This is running on a sub-agent node, - %% so sent skip it + %% so skip it ok end; _ -> % Not on this node diff --git a/lib/snmp/src/agent/snmpa_conf.erl b/lib/snmp/src/agent/snmpa_conf.erl index 4b88eb69f7..c17a6abbd7 100644 --- a/lib/snmp/src/agent/snmpa_conf.erl +++ b/lib/snmp/src/agent/snmpa_conf.erl @@ -424,7 +424,8 @@ target_addr_entry(Name, EngineId, TMask) -> target_addr_entry(Name, Ip, 162, TagList, - ParamsName, EngineId, TMask, 2048). + ParamsName, EngineId, + TMask, 2048). target_addr_entry(Name, Ip, @@ -435,7 +436,8 @@ target_addr_entry(Name, TMask, MaxMessageSize) -> target_addr_entry(Name, Ip, Udp, 1500, 3, TagList, - ParamsName, EngineId, TMask, MaxMessageSize). + ParamsName, EngineId, + TMask, MaxMessageSize). target_addr_entry(Name, Ip, @@ -448,7 +450,8 @@ target_addr_entry(Name, TMask, MaxMessageSize) -> target_addr_entry(Name, snmp_target_mib:default_domain(), Ip, Udp, - Timeout, RetryCount, TagList, ParamsName, + Timeout, RetryCount, TagList, + ParamsName, EngineId, TMask, MaxMessageSize). target_addr_entry(Name, diff --git a/lib/snmp/src/agent/snmpa_mpd.erl b/lib/snmp/src/agent/snmpa_mpd.erl index 14f62b12f3..4f50b1a674 100644 --- a/lib/snmp/src/agent/snmpa_mpd.erl +++ b/lib/snmp/src/agent/snmpa_mpd.erl @@ -32,6 +32,7 @@ -include("SNMP-MPD-MIB.hrl"). -include("SNMPv2-TM.hrl"). -include("SNMP-FRAMEWORK-MIB.hrl"). +-include("TRANSPORT-ADDRESS-MIB.hrl"). -define(VMODULE,"MPD"). -include("snmp_verbosity.hrl"). @@ -981,12 +982,15 @@ generate_discovery_msg2(NoteStore, Pdu, discovery_note_timeout(Timeout) -> (Timeout div 100) + 1. -generate_discovery_msg(NoteStore, {?snmpUDPDomain, [A,B,C,D,U1,U2]}, +generate_discovery_msg(NoteStore, {TDomain, TAddress}, Pdu, ScopedPduBytes, ContextEngineID, ManagerEngineID, SecModel, SecName, SecLevelFlag, InitialUserName, ContextName, Timeout) -> + + {ok, {_Domain, Address}} = transform_taddr(TDomain, TAddress), + %% 7.1.7 ?vdebug("generate_discovery_msg -> 7.1.7 (~w)", [ManagerEngineID]), MsgID = generate_msg_id(), @@ -1027,7 +1031,7 @@ generate_discovery_msg(NoteStore, {?snmpUDPDomain, [A,B,C,D,U1,U2]}, %% Log(Packet), inc_snmp_out_vars(Pdu), ?vdebug("generate_discovery_msg -> done", []), - {Packet, {{A,B,C,D}, U1 bsl 8 + U2}}; + {Packet, Address}; Error -> throw(Error) @@ -1057,6 +1061,34 @@ generate_sec_discovery_msg(Message, SecModule, end. +transform_taddr(?snmpUDPDomain, TAddress) -> + transform_taddr(?transportDomainUdpIpv4, TAddress); +transform_taddr(?transportDomainUdpIpv4, [A, B, C, D, P1, P2]) -> + Domain = transportDomainUdpIpv4, + Addr = {A,B,C,D}, + Port = P1 bsl 8 + P2, + Address = {Addr, Port}, + {ok, {Domain, Address}}; +transform_taddr(?transportDomainUdpIpv4, BadAddr) -> + {error, {bad_transportDomainUdpIpv4_address, BadAddr}}; +transform_taddr(?transportDomainUdpIpv6, + [A1, A2, A3, A4, A5, A6, A7, A8, P1, P2]) -> + Domain = transportDomainUdpIpv6, + Addr = {A1, A2, A3, A4, A5, A6, A7, A8}, + Port = P1 bsl 8 + P2, + Address = {Addr, Port}, + {ok, {Domain, Address}}; +transform_taddr(?transportDomainUdpIpv6, BadAddr) -> + {error, {bad_transportDomainUdpIpv6_address, BadAddr}}; +transform_taddr(BadTDomain, TAddress) -> + case lists:member(BadTDomain, snmp_conf:all_tdomains()) of + true -> + {error, {unsupported_tdomain, BadTDomain, TAddress}}; + false -> + {error, {unknown_tdomain, BadTDomain, TAddress}} + end. + + process_taddrs(Dests) -> ?vtrace("process_taddrs -> entry with" "~n Dests: ~p", [Dests]), @@ -1066,46 +1098,44 @@ process_taddrs([], Acc) -> ?vtrace("process_taddrs -> entry when done with" "~n Acc: ~p", [Acc]), lists:reverse(Acc); - + %% v3 -process_taddrs([{{?snmpUDPDomain, [A,B,C,D,U1,U2]}, SecData} | T], Acc) -> +process_taddrs([{{TDomain, TAddress}, SecData} | T], Acc) -> ?vtrace("process_taddrs -> entry when v3 with" - "~n A: ~p" - "~n B: ~p" - "~n C: ~p" - "~n D: ~p" - "~n U1: ~p" - "~n U2: ~p" - "~n SecData: ~p", [A, B, C, D, U1, U2, SecData]), - Entry = {{snmpUDPDomain, {{A,B,C,D}, U1 bsl 8 + U2}}, SecData}, - process_taddrs(T, [Entry | Acc]); -%% Bad v3 -process_taddrs([{{TDomain, TAddr}, _SecData} | T], Acc) -> - ?vtrace("process_taddrs -> entry when bad v3 with" - "~n TDomain: ~p" - "~n TAddr: ~p", [TDomain, TAddr]), - user_err("Bad TDomain/TAddr: ~w/~w", [TDomain, TAddr]), - process_taddrs(T, Acc); + "~n TDomain: ~p" + "~n TAddress: ~p" + "~n SecData: ~p", [TDomain, TAddress, SecData]), + case transform_taddr(TDomain, TAddress) of + {ok, DestAddr} -> + ?vtrace("process_taddrs -> transformed: " + "~n DestAddr: ~p", [DestAddr]), + Entry = {DestAddr, SecData}, + process_taddrs(T, [Entry | Acc]); + {error, Reason} -> + ?vinfo("Failed transforming v3 domain and address" + "~n Reason: ~p", [Reason]), + user_err("Bad TDomain/TAddress: ~w/~w", [TDomain, TAddress]), + process_taddrs(T, Acc) + end; %% v1 & v2 -process_taddrs([{?snmpUDPDomain, [A,B,C,D,U1,U2]} | T], Acc) -> +process_taddrs([{TDomain, TAddress} | T], Acc) -> ?vtrace("process_taddrs -> entry when v1/v2 with" - "~n A: ~p" - "~n B: ~p" - "~n C: ~p" - "~n D: ~p" - "~n U1: ~p" - "~n U2: ~p", [A, B, C, D, U1, U2]), - Entry = {snmpUDPDomain, {{A,B,C,D}, U1 bsl 8 + U2}}, - process_taddrs(T, [Entry | Acc]); -%% Bad v1 or v2 -process_taddrs([{TDomain, TAddr} | T], Acc) -> - ?vtrace("process_taddrs -> entry when bad v1/v2 with" - "~n TDomain: ~p" - "~n TAddr: ~p", [TDomain, TAddr]), - user_err("Bad TDomain/TAddr: ~w/~w", [TDomain, TAddr]), - process_taddrs(T, Acc); + "~n TDomain: ~p" + "~n TAddress: ~p", [TDomain, TAddress]), + case transform_taddr(TDomain, TAddress) of + {ok, DestAddr} -> + ?vtrace("process_taddrs -> transformed: " + "~n DestAddr: ~p", [DestAddr]), + Entry = DestAddr, + process_taddrs(T, [Entry | Acc]); + {error, Reason} -> + ?vinfo("Failed transforming v1/v2 domain and address: " + "~n Reason: ~p", [Reason]), + user_err("Bad TDomain/TAddress: ~w/~w", [TDomain, TAddress]), + process_taddrs(T, Acc) + end; process_taddrs(Crap, Acc) -> - throw({error, {taddrs_crap, Crap, Acc}}). + throw({error, {bad_taddrs, Crap, Acc}}). mk_v1_v2_packet_list(To, Packet, Len, Pdu) -> diff --git a/lib/snmp/src/app/snmp.appup.src b/lib/snmp/src/app/snmp.appup.src index 5deb40be0f..8e1855b4df 100644 --- a/lib/snmp/src/app/snmp.appup.src +++ b/lib/snmp/src/app/snmp.appup.src @@ -22,82 +22,82 @@ %% ----- U p g r a d e ------------------------------------------------------- [ + {"4.20.1", + [ + {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, []}, + {load_module, snmpm, soft_purge, soft_purge, + [snmpm_server, snmpm_config, snmp_config]}, + {load_module, snmp_conf, soft_purge, soft_purge, []}, + {load_module, snmp_config, soft_purge, soft_purge, []}, + {load_module, snmpm_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config, snmpm_config]}, + {load_module, snmpa_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, + {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_config]}, + {update, snmpa_agent, soft, soft_purge, soft_purge, [snmpa_mpd]}, + {update, snmpm_config, soft, soft_purge, soft_purge, [snmp_conf]}, + {update, snmpm_server, soft, soft_purge, soft_purge, + [snmpm_net_if, snmpm_mpd, snmpm_config]}, + {update, snmpm_net_if, soft, soft_purge, soft_purge, + [snmp_conf, snmpm_mpd, snmpm_config]} + ] + }, + {"4.20", + [ + {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, []}, + {load_module, snmp_target_mib, soft_purge, soft_purge, [snmp_conf]}, + {load_module, snmpm, soft_purge, soft_purge, + [snmpm_server, snmpm_config, snmp_config]}, + {load_module, snmp_conf, soft_purge, soft_purge, []}, + {load_module, snmp_config, soft_purge, soft_purge, []}, + {load_module, snmpm_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config, snmpm_config]}, + {load_module, snmpa_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, + {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_config]}, + {update, snmpa_agent, soft, soft_purge, soft_purge, [snmpa_mpd]}, + {update, snmpm_config, soft, soft_purge, soft_purge, [snmp_conf]}, + {update, snmpm_server, soft, soft_purge, soft_purge, + [snmpm_net_if, snmpm_mpd, snmpm_config]}, + {update, snmpm_net_if, soft, soft_purge, soft_purge, + [snmp_conf, snmpm_mpd, snmpm_config]} + ] + }, {"4.19", [ {load_module, snmpa, soft_purge, soft_purge, []}, - {load_module, snmpm, soft_purge, soft_purge, [snmpm_server]}, + {load_module, snmpm, soft_purge, soft_purge, + [snmpm_server, snmpm_config, snmp_config]}, {load_module, snmpa_usm, soft_purge, soft_purge, []}, {load_module, snmpm_usm, soft_purge, soft_purge, []}, {load_module, snmp_log, soft_purge, soft_purge, []}, {load_module, snmp_pdus, soft_purge, soft_purge, []}, {load_module, snmp_conf, soft_purge, soft_purge, []}, - {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_conf]}, + {load_module, snmpa_conf, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, {load_module, snmp_misc, soft_purge, soft_purge, []}, {load_module, snmp_config, soft_purge, soft_purge, []}, - {load_module, snmpa_mpd, soft_purge, soft_purge, [snmp_conf]}, + {load_module, snmpa_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, + {load_module, snmpm_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config, snmpm_config]}, {load_module, snmpa_trap, soft_purge, soft_purge, [snmpa_mpd, snmp_notification_mib, snmp_target_mib, snmpa_net_if]}, {load_module, snmpa_acm, soft_purge, soft_purge, [snmp_conf, snmpa_mpd, snmp_target_mib]}, {load_module, snmpa_conf, soft_purge, soft_purge, - [snmp_notification_mib]}, + [snmp_config, snmp_notification_mib]}, + {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, []}, {load_module, snmp_notification_mib, soft_purge, soft_purge, [snmp_conf, snmp_target_mib]}, {load_module, snmp_community_mib, soft_purge, soft_purge, []}, {load_module, snmp_target_mib, soft_purge, soft_purge, [snmp_conf]}, - {update, snmpm_net_if, soft, soft_purge, soft_purge, []}, - {update, snmpm_server, soft, soft_purge, soft_purge, [snmpm_net_if]}, - {update, snmpa_net_if, soft, soft_purge, soft_purge, - [snmp_conf, snmpa_mpd]}, - {update, snmpa_agent, soft, soft_purge, soft_purge, - [snmpa_acm, snmpa_mpd, snmpa_trap]} - ] - }, - {"4.18", - [ - {load_module, snmpa, soft_purge, soft_purge, []}, - {load_module, snmpm, soft_purge, soft_purge, [snmpm_server]}, - {load_module, snmpa_usm, soft_purge, soft_purge, []}, - {load_module, snmpm_usm, soft_purge, soft_purge, []}, - {load_module, snmp_misc, soft_purge, soft_purge, []}, - {load_module, snmp_log, soft_purge, soft_purge, []}, - {load_module, snmp_pdus, soft_purge, soft_purge, []}, - {load_module, snmp_conf, soft_purge, soft_purge, []}, - {load_module, snmp_config, soft_purge, soft_purge, [snmp_conf]}, - {load_module, snmpa_conf, soft_purge, soft_purge, []}, - {load_module, snmpa_mpd, soft_purge, soft_purge, [snmp_conf]}, - {load_module, snmpa_vacm, soft_purge, soft_purge, []}, - {load_module, snmpa_trap, soft_purge, soft_purge, - [snmpa_mpd, snmp_notification_mib, snmp_target_mib, snmpa_net_if]}, - {load_module, snmpa_acm, soft_purge, soft_purge, - [snmp_conf, snmpa_mpd, snmp_target_mib]}, - {load_module, snmpa, soft_purge, soft_purge, - [snmp_community_mib, - snmp_framework_mib, - snmp_standard_mib, - snmp_target_mib, - snmp_user_based_sm_mib, - snmp_view_based_acm_mib]}, - {load_module, snmp_notification_mib, soft_purge, soft_purge, - [snmp_conf, snmp_target_mib, snmpa_mib_lib]}, - {load_module, snmp_community_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_framework_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_standard_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_target_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_user_based_sm_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, - [snmpa_mib_lib, snmpa_vacm]}, - {load_module, snmpa_mib_lib, soft_purge, soft_purge, []}, - - {update, snmpm_net_if, soft, soft_purge, soft_purge, []}, - {update, snmpm_server, soft, soft_purge, soft_purge, [snmpm_net_if]}, - + {update, snmpm_net_if, soft, soft_purge, soft_purge, + [snmp_conf, snmpm_mpd, snmpm_config]}, + {update, snmpm_config, soft, soft_purge, soft_purge, [snmp_conf]}, + {update, snmpm_server, soft, soft_purge, soft_purge, + [snmpm_net_if, snmpm_mpd, snmpm_config]}, {update, snmpa_net_if, soft, soft_purge, soft_purge, [snmp_conf, snmpa_mpd]}, {update, snmpa_agent, soft, soft_purge, soft_purge, @@ -109,84 +109,82 @@ %% ------D o w n g r a d e --------------------------------------------------- [ + {"4.20.1", + [ + {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, []}, + {load_module, snmpm, soft_purge, soft_purge, + [snmpm_server, snmpm_config, snmp_config]}, + {load_module, snmp_conf, soft_purge, soft_purge, []}, + {load_module, snmp_config, soft_purge, soft_purge, []}, + {load_module, snmpm_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config, snmpm_config]}, + {load_module, snmpa_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, + {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_config]}, + {update, snmpa_agent, soft, soft_purge, soft_purge, [snmpa_mpd]}, + {update, snmpm_config, soft, soft_purge, soft_purge, [snmp_conf]}, + {update, snmpm_server, soft, soft_purge, soft_purge, + [snmpm_net_if, snmpm_mpd, snmpm_config]}, + {update, snmpm_net_if, soft, soft_purge, soft_purge, + [snmp_conf, snmpm_mpd, snmpm_config]} + ] + }, + {"4.20", + [ + {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, []}, + {load_module, snmp_target_mib, soft_purge, soft_purge, [snmp_conf]}, + {load_module, snmpm, soft_purge, soft_purge, + [snmpm_server, snmpm_config, snmp_config]}, + {load_module, snmp_conf, soft_purge, soft_purge, []}, + {load_module, snmp_config, soft_purge, soft_purge, []}, + {load_module, snmpm_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config, snmpm_config]}, + {load_module, snmpa_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, + {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_config]}, + {update, snmpa_agent, soft, soft_purge, soft_purge, [snmpa_mpd]}, + {update, snmpm_config, soft, soft_purge, soft_purge, [snmp_conf]}, + {update, snmpm_server, soft, soft_purge, soft_purge, + [snmpm_net_if, snmpm_mpd, snmpm_config]}, + {update, snmpm_net_if, soft, soft_purge, soft_purge, + [snmp_conf, snmpm_mpd, snmpm_config]} + ] + }, {"4.19", [ {load_module, snmpa, soft_purge, soft_purge, []}, - {load_module, snmpm, soft_purge, soft_purge, [snmpm_server]}, + {load_module, snmpm, soft_purge, soft_purge, + [snmpm_server, snmpm_config, snmp_config]}, {load_module, snmpa_usm, soft_purge, soft_purge, []}, {load_module, snmpm_usm, soft_purge, soft_purge, []}, {load_module, snmp_log, soft_purge, soft_purge, []}, {load_module, snmp_pdus, soft_purge, soft_purge, []}, {load_module, snmp_conf, soft_purge, soft_purge, []}, - {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_conf]}, + {load_module, snmpa_conf, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, {load_module, snmp_misc, soft_purge, soft_purge, []}, {load_module, snmp_config, soft_purge, soft_purge, []}, - {load_module, snmpa_mpd, soft_purge, soft_purge, [snmp_conf]}, + {load_module, snmpa_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config]}, + {load_module, snmpm_mpd, soft_purge, soft_purge, + [snmp_conf, snmp_config, snmpm_config]}, {load_module, snmpa_trap, soft_purge, soft_purge, [snmpa_mpd, snmp_notification_mib, snmp_target_mib, snmpa_net_if]}, {load_module, snmpa_acm, soft_purge, soft_purge, [snmp_conf, snmpa_mpd, snmp_target_mib]}, {load_module, snmpa_conf, soft_purge, soft_purge, - [snmp_notification_mib]}, + [snmp_config, snmp_notification_mib]}, + {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, []}, {load_module, snmp_notification_mib, soft_purge, soft_purge, [snmp_conf, snmp_target_mib]}, {load_module, snmp_community_mib, soft_purge, soft_purge, []}, {load_module, snmp_target_mib, soft_purge, soft_purge, [snmp_conf]}, - - {update, snmpm_net_if, soft, soft_purge, soft_purge, []}, - {update, snmpm_server, soft, soft_purge, soft_purge, [snmpm_net_if]}, - - {update, snmpa_net_if, soft, soft_purge, soft_purge, - [snmp_conf, snmpa_mpd]}, - {update, snmpa_agent, soft, soft_purge, soft_purge, - [snmpa_acm, snmpa_mpd, snmpa_trap]} - ] - }, - {"4.18", - [ - {load_module, snmpa, soft_purge, soft_purge, []}, - {load_module, snmpm, soft_purge, soft_purge, [snmpm_server]}, - {load_module, snmpa_usm, soft_purge, soft_purge, []}, - {load_module, snmpm_usm, soft_purge, soft_purge, []}, - {load_module, snmp_misc, soft_purge, soft_purge, []}, - {load_module, snmp_log, soft_purge, soft_purge, []}, - {load_module, snmp_pdus, soft_purge, soft_purge, []}, - {load_module, snmp_conf, soft_purge, soft_purge, []}, - {load_module, snmpa_conf, soft_purge, soft_purge, [snmp_conf]}, - {load_module, snmp_config, soft_purge, soft_purge, []}, - {load_module, snmpa_mpd, soft_purge, soft_purge, [snmp_conf]}, - {load_module, snmpa_vacm, soft_purge, soft_purge, []}, - {load_module, snmpa_trap, soft_purge, soft_purge, - [snmpa_mpd, snmp_notification_mib, snmp_target_mib, snmpa_net_if]}, - {load_module, snmpa_acm, soft_purge, soft_purge, - [snmp_conf, snmpa_mpd, snmp_target_mib]}, - {load_module, snmpa, soft_purge, soft_purge, - [snmp_community_mib, - snmp_framework_mib, - snmp_standard_mib, - snmp_target_mib, - snmp_user_based_sm_mib, - snmp_view_based_acm_mib]}, - {load_module, snmp_notification_mib, soft_purge, soft_purge, - [snmp_conf, snmp_target_mib, snmpa_mib_lib]}, - {load_module, snmp_community_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_framework_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_standard_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_target_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_user_based_sm_mib, soft_purge, soft_purge, - [snmpa_mib_lib]}, - {load_module, snmp_view_based_acm_mib, soft_purge, soft_purge, - [snmpa_mib_lib, snmpa_vacm]}, - {load_module, snmpa_mib_lib, soft_purge, soft_purge, []}, - - {update, snmpm_net_if, soft, soft_purge, soft_purge, []}, - {update, snmpm_server, soft, soft_purge, soft_purge, [snmpm_net_if]}, - + {update, snmpm_net_if, soft, soft_purge, soft_purge, + [snmp_conf, snmpm_mpd, snmpm_config]}, + {update, snmpm_config, soft, soft_purge, soft_purge, [snmp_conf]}, + {update, snmpm_server, soft, soft_purge, soft_purge, + [snmpm_net_if, snmpm_mpd, snmpm_config]}, {update, snmpa_net_if, soft, soft_purge, soft_purge, [snmp_conf, snmpa_mpd]}, {update, snmpa_agent, soft, soft_purge, soft_purge, diff --git a/lib/snmp/src/compile/Makefile b/lib/snmp/src/compile/Makefile index 0ceaf276a6..627af6f185 100644 --- a/lib/snmp/src/compile/Makefile +++ b/lib/snmp/src/compile/Makefile @@ -45,11 +45,10 @@ RELSYSDIR = $(RELEASE_PATH)/lib/snmp-$(VSN) include modules.mk -ESCRIPT_BIN = $(ESCRIPT_SRC:%.src=$(BIN)/%) - -ERL_FILES = $(MODULES:%=%.erl) - -TARGET_FILES = $(MODULES:%=$(EBIN)/%.$(EMULATOR)) $(ESCRIPT_BIN) +ESCRIPT_BIN = $(ESCRIPT_SRC:%.src=$(BIN)/%) +ERL_FILES = $(MODULES:%=%.erl) +EBIN_FILES = $(MODULES:%=$(EBIN)/%.$(EMULATOR)) +TARGET_FILES = $(EBIN_FILES) $(ESCRIPT_BIN) GENERATED_PARSER = $(PARSER_MODULE:%=%.erl) @@ -125,7 +124,7 @@ release_spec: opt $(INSTALL_DIR) $(RELSYSDIR)/src/compiler $(INSTALL_DATA) $(ESCRIPT_SRC) $(PARSER_SRC) $(ERL_FILES) $(INTERNAL_HRL_FILES) $(RELSYSDIR)/src/compiler $(INSTALL_DIR) $(RELSYSDIR)/ebin - $(INSTALL_DATA) $(TARGET_FILES) $(RELSYSDIR)/ebin + $(INSTALL_DATA) $(EBIN_FILES) $(RELSYSDIR)/ebin $(INSTALL_DIR) $(RELSYSDIR)/bin $(INSTALL_SCRIPT) $(ESCRIPT_BIN) $(RELSYSDIR)/bin diff --git a/lib/snmp/src/compile/snmpc.erl b/lib/snmp/src/compile/snmpc.erl index 195c238184..5e6b81f1ec 100644 --- a/lib/snmp/src/compile/snmpc.erl +++ b/lib/snmp/src/compile/snmpc.erl @@ -108,6 +108,7 @@ compile(FileName) -> %% {i, [import_dir_string()]} ["./"] %% {il, [import_lib_dir_string()]} [] %% {warnings, bool()} true +%% warnings_as_errors %% {outdir, string()} "./" %% description %% reference @@ -199,6 +200,8 @@ get_options([reference|Opts], Formats, Args) -> get_options(Opts, ["~n reference"|Formats], Args); get_options([{warnings, Val}|Opts], Formats, Args) -> get_options(Opts, ["~n warnings: ~w"|Formats], [Val|Args]); +get_options([warnings_as_errors|Opts], Formats, Args) -> + get_options(Opts, ["~n warnings_as_errors"|Formats], Args); get_options([{verbosity, Val}|Opts], Formats, Args) -> get_options(Opts, ["~n verbosity: ~w"|Formats], [Val|Args]); get_options([imports|Opts], Formats, Args) -> @@ -261,6 +264,8 @@ check_options([{group_check, Atom} | T]) when is_atom(Atom) -> check_options([{warnings, Bool} | T]) -> check_bool(warnings, Bool), check_options(T); +check_options([warnings_as_errors | T]) -> + check_options(T); check_options([{db, volatile} | T]) -> check_options(T); check_options([{db, persistent} | T]) -> @@ -331,6 +336,9 @@ get_agent_capabilities(Options) -> get_module_compliance(Options) -> get_bool_option(module_compliance, Options). +get_warnings_as_errors(Options) -> + lists:member(warnings_as_errors, Options). + get_relaxed_row_name_assign_check(Options) -> lists:member(relaxed_row_name_assign_check, Options). @@ -409,6 +417,7 @@ init(From, MibFileName, Options) -> put(reference, get_reference(Options)), put(agent_capabilities, get_agent_capabilities(Options)), put(module_compliance, get_module_compliance(Options)), + put(warnings_as_errors, get_warnings_as_errors(Options)), File = filename:rootname(MibFileName, ".mib"), put(filename, filename:basename(File ++ ".mib")), R = case catch c_impl(File) of diff --git a/lib/snmp/src/compile/snmpc.src b/lib/snmp/src/compile/snmpc.src index 5f9b154bfa..4e91ae9a03 100644 --- a/lib/snmp/src/compile/snmpc.src +++ b/lib/snmp/src/compile/snmpc.src @@ -50,7 +50,8 @@ %% The default verbosity (silence) will be filled in %% during argument processing. verbosity, - warnings = false + warnings = false, + warnings_as_errors = false }). @@ -74,6 +75,7 @@ %% --version %% --verbosity V %% --warnings +%% --wae main(Args) when is_list(Args) -> case (catch process_args(Args)) of ok -> @@ -156,7 +158,8 @@ mk_mib_options(#state{outdir = OutDir, %% The default verbosity (silence) will be filled in %% during argument processing. verbosity = V, - warnings = W}) -> + warnings = W, + warnings_as_errors = WAE}) -> [{outdir, OutDir}, {db, DB}, {i, IDs}, @@ -178,7 +181,8 @@ mk_mib_options(#state{outdir = OutDir, maybe_option(Imp, imports) ++ maybe_option(MI, module_identity) ++ maybe_option(MC, module_compliance) ++ - maybe_option(AC, agent_capabilities). + maybe_option(AC, agent_capabilities) ++ + maybe_option(WE, warnings_as_errors). maybe_option(true, Opt) -> [Opt]; maybe_option(_, _) -> []. @@ -292,6 +296,8 @@ process_args(["--nd"|Args], State) -> process_args(Args, State#state{no_defaults = true}); process_args(["--rrnac"|Args], State) -> process_args(Args, State#state{relaxed_row_name_assigne_check = true}); +process_args(["--wae"|Args], State) -> + process_args(Args, State#state{warnings_as_errors = true}); process_args([MIB], State) -> Ext = filename:extension(MIB), if @@ -371,6 +377,8 @@ usage() -> "~n a warning. " "~n By default it is not included, but if this option is " "~n present it will be. " + "~n --wae - Warnings as errors. " + "~n Indicates that warnings shall be treated as errors. " "~n " "~n", []), halt(1). diff --git a/lib/snmp/src/compile/snmpc_lib.hrl b/lib/snmp/src/compile/snmpc_lib.hrl index 000486e728..35ec9abd03 100644 --- a/lib/snmp/src/compile/snmpc_lib.hrl +++ b/lib/snmp/src/compile/snmpc_lib.hrl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2009. All Rights Reserved. +%% Copyright Ericsson AB 2009-2011. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -20,8 +20,17 @@ -ifndef(snmpc_lib). -define(snmpc_lib, true). --define(vwarning(F, A), ?verbosity(warning, F, A, ignore)). --define(vwarning2(F, A, MibLine), ?verbosity(warning, F, A, MibLine)). +-define(vwarning(F, A), + case get(warnings_as_errors) of + true -> snmpc_lib:error(F, A); + _ -> ?verbosity(warning, F, A, ignore) + end). + +-define(vwarning2(F, A, MibLine), + case get(warnings_as_errors) of + true -> snmpc_lib:error(F, A, MibLine); + _ -> ?verbosity(warning, F, A, MibLine) + end). -define(vinfo(F, A), ?verbosity(info, F, A, ignore)). -define(vinfo2(F, A, MibLine), ?verbosity(info, F, A, MibLine)). -define(vlog(F, A), ?verbosity(log, F, A, ignore)). diff --git a/lib/snmp/src/manager/snmpm.erl b/lib/snmp/src/manager/snmpm.erl index 0d084332de..6d2ac8d747 100644 --- a/lib/snmp/src/manager/snmpm.erl +++ b/lib/snmp/src/manager/snmpm.erl @@ -50,7 +50,7 @@ register_agent/2, register_agent/3, register_agent/4, unregister_agent/2, unregister_agent/3, which_agents/0, which_agents/1, - agent_info/2, update_agent_info/4, + agent_info/2, update_agent_info/3, update_agent_info/4, register_usm_user/3, unregister_usm_user/2, which_usm_users/0, which_usm_users/1, @@ -167,6 +167,7 @@ -include_lib("snmp/include/snmp_types.hrl"). -include("snmpm_atl.hrl"). -include("snmpm_internal.hrl"). +-include("snmp_verbosity.hrl"). -define(DEFAULT_AGENT_PORT, 161). @@ -447,8 +448,11 @@ agent_info(Addr, Port, Item) -> Error end. +update_agent_info(UserId, TargetName, Info) when is_list(Info) -> + snmpm_config:update_agent_info(UserId, TargetName, Info). + update_agent_info(UserId, TargetName, Item, Val) -> - snmpm_config:update_agent_info(UserId, TargetName, Item, Val). + update_agent_info(UserId, TargetName, [{Item, Val}]). %% Backward compatibility functions update_agent_info(UserId, Addr, Port, Item, Val) -> diff --git a/lib/snmp/src/manager/snmpm_config.erl b/lib/snmp/src/manager/snmpm_config.erl index fd6da3e71a..c2e57abddb 100644 --- a/lib/snmp/src/manager/snmpm_config.erl +++ b/lib/snmp/src/manager/snmpm_config.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2004-2010. All Rights Reserved. +%% Copyright Ericsson AB 2004-2011. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -36,7 +36,8 @@ user_info/0, user_info/1, user_info/2, register_agent/3, unregister_agent/2, - agent_info/0, agent_info/2, agent_info/3, update_agent_info/4, + agent_info/0, agent_info/2, agent_info/3, + update_agent_info/3, update_agent_info/4, which_agents/0, which_agents/1, is_known_engine_id/2, @@ -84,7 +85,9 @@ backup/1, - mk_target_name/3 + mk_target_name/3, + + default_transport_domain/0 ]). @@ -127,23 +130,24 @@ %% Macros and Constants: --define(SERVER, ?MODULE). --define(BACKUP_DB, snmpm_config_backup). --define(CONFIG_DB, snmpm_config_db). +-define(SERVER, ?MODULE). +-define(BACKUP_DB, snmpm_config_backup). +-define(CONFIG_DB, snmpm_config_db). -define(DEFAULT_USER, default_user). -define(DEFAULT_AGENT_PORT, 161). --define(IRB_DEFAULT, auto). -%% -define(IRB_DEFAULT, {user, timer:seconds(15)}). +-define(IRB_DEFAULT, auto). +%% -define(IRB_DEFAULT, {user, timer:seconds(15)}). --define(USER_MOD_DEFAULT, snmpm_user_default). --define(USER_DATA_DEFAULT, undefined). +-define(USER_MOD_DEFAULT, snmpm_user_default). +-define(USER_DATA_DEFAULT, undefined). %% -define(DEF_ADDR_TAG, default_addr_tag). -define(DEFAULT_TARGETNAME, default_agent). --define(DEF_PORT_TAG, default_port_tag). +-define(DEF_PORT_TAG, default_port_tag). +-define(SUPPORTED_DOMAINS, [transportDomainUdpIpv4, transportDomainUdpIpv6]). -ifdef(snmp_debug). -define(GS_START_LINK(Opts), @@ -159,6 +163,11 @@ %%%------------------------------------------------------------------- %%% API %%%------------------------------------------------------------------- + +default_transport_domain() -> + transportDomainUdpIpv4. + + start_link(Opts) -> ?d("start_link -> entry with" "~n Opts: ~p", [Opts]), @@ -269,9 +278,10 @@ do_user_info(_UserId, BadItem) -> error({not_found, BadItem}). -%% A target-name constructed in this way is a string with the following +%% A target-name constructed in this way is a string with the following: %% <IP-address>:<Port>-<Version> -%% +%% This is intended for backward compatibility and therefor has +%% only support for IPv4 addresses and *no* other transport domain. mk_target_name(Addr0, Port, Config) when is_list(Config) -> Version = case lists:keysearch(version, 1, Config) of @@ -280,7 +290,6 @@ mk_target_name(Addr0, Port, Config) when is_list(Config) -> false -> select_lowest_supported_version() end, -%% p("mk_target_name -> Version: ~p", [Version]), case normalize_address(Addr0) of {A, B, C, D} -> lists:flatten( @@ -308,57 +317,99 @@ select_lowest_supported_version([H|T], Versions) -> end. -register_agent(UserId, _TargetName, _Config) when (UserId =:= user_id) -> +register_agent(UserId, _TargetName, _Config0) when (UserId =:= user_id) -> {error, {bad_user_id, UserId}}; -register_agent(UserId, TargetName, Config) +register_agent(UserId, TargetName, Config0) when (is_list(TargetName) andalso (length(TargetName) > 0) andalso - is_list(Config)) -> + is_list(Config0)) -> -%% p("register_agent -> entry with" -%% "~n UserId: ~p" -%% "~n TargetName: ~p" -%% "~n Config: ~p", [UserId, TargetName, Config]), + ?vtrace("register_agent -> entry with" + "~n UserId: ~p" + "~n TargetName: ~p" + "~n Config0: ~p", [UserId, TargetName, Config0]), %% Check: %% 1) That the mandatory configs are present - %% 2) That the illegal config user_id (used internally) is - %% not present + %% 2) That no illegal config, e.g. user_id (used internally), + %% is not present %% 3) Check that there are no invalid or erroneous configs - %% 4) Chack that the manager is capable to use the selected version - case verify_agent_config(Config) of - ok -> + %% 4) Check that the manager is capable of using the selected version + case verify_agent_config(Config0) of + {ok, Config} -> call({register_agent, UserId, TargetName, Config}); Error -> Error end. -verify_agent_config(Conf) -> - ok = verify_mandatory(Conf, [engine_id, address, reg_type]), - case verify_invalid(Conf, [user_id]) of - ok -> - case verify_agent_config2(Conf) of - ok -> - {ok, Vsns} = system_info(versions), - Vsn = - case lists:keysearch(version, 1, Conf) of - {value, {version, V}} -> - V; - false -> - v1 - end, - case lists:member(Vsn, Vsns) of - true -> - ok; - false -> - {error, {version_not_supported_by_manager, Vsn, Vsns}} - end - end; - Error -> +verify_agent_config(Conf0) -> + try + begin + verify_mandatory(Conf0, [engine_id, address, reg_type]), + verify_invalid(Conf0, [user_id]), + Conf = verify_agent_config3(Conf0), + Vsns = versions(), + Vsn = which_version(Conf), + verify_version(Vsn, Vsns), + {ok, Conf} + end + catch + throw:Error -> Error end. +versions() -> + case system_info(versions) of + {ok, Vsns} -> + Vsns; + {error, _} = ERROR -> + throw(ERROR) + end. + +which_version(Conf) -> + case lists:keysearch(version, 1, Conf) of + {value, {version, V}} -> + V; + false -> + v1 + end. + +verify_version(Vsn, Vsns) -> + case lists:member(Vsn, Vsns) of + true -> + ok; + false -> + Reason = {version_not_supported_by_manager, Vsn, Vsns}, + throw({error, Reason}) + end. + +verify_agent_config3(Conf0) -> + %% Fix (transport) address and domain + {TDomain, Conf1} = + case lists:keysearch(tdomain, 1, Conf0) of + {value, {tdomain, Dom}} -> + {Dom, Conf0}; + false -> + Dom = default_transport_domain(), + {Dom, [{tdomain, Dom} | Conf0]} + end, + Conf2 = case lists:keysearch(address, 1, Conf1) of + {value, {address, Address}} -> + lists:keyreplace(address, 1, Conf1, + {address, {TDomain, Address}}); + false -> + %% This is a mandatory config option, + %% a later test will detect this + Conf1 + end, + case verify_agent2(Conf2) of + {ok, Conf} -> + Conf; + {error, _} = ERROR -> + throw(ERROR) + end. + verify_agent_config2(Conf) -> verify_agent2(Conf). @@ -366,6 +417,7 @@ verify_agent_config2(Conf) -> unregister_agent(UserId, TargetName) -> call({unregister_agent, UserId, TargetName}). +%% This is the old style agent unregistration (using Addr and Port). unregister_agent(UserId, Addr0, Port) -> Addr = normalize_address(Addr0), case do_agent_info(Addr, Port, target_name) of @@ -421,17 +473,51 @@ which_agents(UserId) -> Agents = ets:match(snmpm_agent_table, Pat), [TargetName || [TargetName] <- Agents]. - -update_agent_info(UserId, TargetName, Item, Val0) - when (Item =/= user_id) -> - case (catch verify_val(Item, Val0)) of - {ok, Val} -> - call({update_agent_info, UserId, TargetName, Item, Val}); - Error -> + +verify_agent_info(TargetName, Info0) -> + try + begin + verify_invalid(Info0, [user_id]), + %% Check if address is part of the list and + %% if so update it with the domain info. + Info = + case lists:keysearch(address, 1, Info0) of + {value, {address, Addr}} -> + %% If domain is part of the info, then use it. + %% If not, lookup what is already stored for + %% this agent and use that. + Domain = + case lists:keysearch(tdomain, 1, Info0) of + {value, {tdomain, Dom}} -> + Dom; + false -> + {ok, Dom} = + agent_info(TargetName, tdomain), + Dom + end, + Addr2 = {Domain, Addr}, + lists:keyreplace(address, 1, Info0, {address, Addr2}); + false -> + Info0 + end, + verify_agent2(Info) + end + catch + throw:Error -> Error end. -%% Backward compatibillity +update_agent_info(UserId, TargetName, Info) -> + call({update_agent_info, UserId, TargetName, Info}). + +%% <BACKWARD-COMPAT-2> +%% This is wrapped in the interface module, so this function is +%% only here to catch code-upgrade problems. +update_agent_info(UserId, TargetName, Item, Val) -> + update_agent_info(UserId, TargetName, [{Item, Val}]). +%% </BACKWARD-COMPAT-2> + +%% <BACKWARD-COMPAT-1> update_agent_info(UserId, Addr, Port, Item, Val) -> case agent_info(Addr, Port, target_name) of {ok, TargetName} -> @@ -439,6 +525,7 @@ update_agent_info(UserId, Addr, Port, Item, Val) -> Error -> Error end. +%% </BACKWARD-COMPAT-1> is_known_engine_id(EngineID, TargetName) -> case agent_info(TargetName, engine_id) of @@ -650,22 +737,14 @@ unregister_usm_user(EngineID, Name) call({unregister_usm_user, EngineID, Name}). verify_usm_user_config(EngineID, Name, Config) -> - %% case verify_mandatory(Config, []) of - %% ok -> - %% case verify_invalid(Config, [engine_id, name]) of - %% ok -> - %% verify_usm_user_config2(EngineID, Name, Config); - %% Error -> - %% Error - %% end; - %% Error -> - %% Error - %% end. - ok = verify_mandatory(Config, []), - case verify_invalid(Config, [engine_id, name]) of - ok -> - verify_usm_user_config2(EngineID, Name, Config); - Error -> + try + begin + verify_mandatory(Config, []), + verify_invalid(Config, [engine_id, name]), + verify_usm_user_config2(EngineID, Name, Config) + end + catch + throw:Error -> Error end. @@ -1590,6 +1669,7 @@ check_agent_config2(Agent) -> throw(Err) end. +%% For backward compatibility check_agent_config({UserId, TargetName, Community, @@ -1597,10 +1677,27 @@ check_agent_config({UserId, EngineId, Timeout, MaxMessageSize, Version, SecModel, SecName, SecLevel}) -> + TDomain = default_transport_domain(), + check_agent_config({UserId, + TargetName, + Community, + TDomain, Ip, Port, + EngineId, + Timeout, MaxMessageSize, + Version, SecModel, SecName, SecLevel}); + +check_agent_config({UserId, + TargetName, + Community, + TDomain, Ip, Port, + EngineId, + Timeout, MaxMessageSize, + Version, SecModel, SecName, SecLevel}) -> ?vtrace("check_agent_config -> entry with" "~n UserId: ~p" "~n TargetName: ~p" "~n Community: ~p" + "~n TDomain: ~p" "~n Ip: ~p" "~n Port: ~p" "~n EngineId: ~p" @@ -1610,15 +1707,16 @@ check_agent_config({UserId, "~n SecModel: ~p" "~n SecName: ~p" "~n SecLevel: ~p", - [UserId, TargetName, Community, Ip, Port, + [UserId, TargetName, Community, + TDomain, Ip, Port, EngineId, Timeout, MaxMessageSize, Version, SecModel, SecName, SecLevel]), - Addr = normalize_address(Ip), + Addr = normalize_address(TDomain, Ip), ?vtrace("check_agent_config -> Addr: ~p", [Addr]), Agent = {UserId, TargetName, Community, - Addr, Port, + TDomain, Addr, Port, EngineId, Timeout, MaxMessageSize, Version, SecModel, SecName, SecLevel}, @@ -1644,6 +1742,7 @@ init_agent_config({UserId, TargetName, Config}) -> end. +%% For backward compatibility verify_agent({UserId, TargetName, Comm, @@ -1651,48 +1750,68 @@ verify_agent({UserId, EngineId, Timeout, MMS, Version, SecModel, SecName, SecLevel}) -> - ?vtrace("verify_agent -> entry with" + TDomain = default_transport_domain(), + verify_agent({UserId, + TargetName, + Comm, + TDomain, Ip, Port, + EngineId, + Timeout, MMS, + Version, SecModel, SecName, SecLevel}); + +verify_agent({UserId, + TargetName, + Comm, + TDomain, Ip, Port, + EngineId, + Timeout, MMS, + Version, SecModel, SecName, SecLevel}) -> + ?vdebug("verify_agent -> entry with" "~n UserId: ~p" "~n TargetName: ~p", [UserId, TargetName]), snmp_conf:check_string(TargetName, {gt, 0}), - case verify_val(address, Ip) of - {ok, Addr} -> - snmp_conf:check_integer(Port, {gt, 0}), - Conf = - [{reg_type, target_name}, - {address, Addr}, - {port, Port}, - {community, Comm}, - {engine_id, EngineId}, - {timeout, Timeout}, - {max_message_size, MMS}, - {version, Version}, - {sec_model, SecModel}, - {sec_name, SecName}, - {sec_level, SecLevel} - ], - case verify_agent2(Conf) of - ok -> - {UserId, TargetName, Conf, Version}; - Err -> - throw(Err) - end; - - Error -> - ?vlog("verify_agent -> failed: ~n ~p", [Error]), - throw(Error) + snmp_conf:check_integer(Port, {gt, 0}), + %% Note that the order of Conf *is* important. + %% Some properties may depend on others, so that + %% in order to verify one property, another must + %% be already verified (and present). An example + %% of this is the property 'address', for which + %% the property tdomain is needed. + Conf0 = + [{reg_type, target_name}, + {tdomain, TDomain}, + %% This should be taddress, but what the*... + {address, {TDomain, Ip}}, + {port, Port}, + {community, Comm}, + {engine_id, EngineId}, + {timeout, Timeout}, + {max_message_size, MMS}, + {version, Version}, + {sec_model, SecModel}, + {sec_name, SecName}, + {sec_level, SecLevel} + ], + case verify_agent2(Conf0) of + {ok, Conf} -> + {UserId, TargetName, Conf, Version}; + Err -> + throw(Err) end. -verify_agent2([]) -> - ok; -verify_agent2([{Item, Val}|Items]) -> - case verify_val(Item, Val) of - {ok, _Val} -> - verify_agent2(Items); +verify_agent2(Conf) -> + verify_agent2(Conf, []). + +verify_agent2([], VerifiedConf) -> + {ok, VerifiedConf}; +verify_agent2([{Item, Val0}|Items], VerifiedConf) -> + case verify_val(Item, Val0) of + {ok, Val} -> + verify_agent2(Items, [{Item, Val} | VerifiedConf]); Err -> Err end; -verify_agent2([Bad|_]) -> +verify_agent2([Bad|_], _VerifiedConf) -> {error, {bad_agent_config, Bad}}. @@ -1708,14 +1827,28 @@ read_users_config_file(Dir) -> check_user_config({Id, Mod, Data}) -> + ?vtrace("check_user_config -> entry with" + "~n Id: ~p" + "~n Mod: ~p" + "~n Data: ~p", [Id, Mod, Data]), check_user_config({Id, Mod, Data, []}); -check_user_config({Id, Mod, _Data, DefaultAgentConfig} = User) +check_user_config({Id, Mod, Data, DefaultAgentConfig} = _User) when (Id =/= ?DEFAULT_USER) andalso is_list(DefaultAgentConfig) -> + ?vtrace("check_user_config -> entry with" + "~n Id: ~p" + "~n Mod: ~p" + "~n Data: ~p" + "~n DefaultAgentConfig: ~p", + [Id, Mod, Data, DefaultAgentConfig]), case (catch verify_user_behaviour(Mod)) of ok -> + ?vtrace("check_user_config -> user behaviour verified", []), case verify_user_agent_config(DefaultAgentConfig) of - ok -> - {ok, User}; + {ok, DefAgentConf} -> + ?vtrace("check_user_config -> " + "user agent (default) config verified", []), + User2 = {Id, Mod, Data, DefAgentConf}, + {ok, User2}; {error, Reason} -> error({bad_default_agent_config, Reason}) end; @@ -1756,16 +1889,16 @@ verify_user({Id, UserMod, UserData}) -> verify_user({Id, UserMod, UserData, DefaultAgentConfig}) when (Id =/= ?DEFAULT_USER) andalso is_list(DefaultAgentConfig) -> ?d("verify_user -> entry with" - "~n Id: ~p" - "~n UserMod: ~p" - "~n UserData: ~p" + "~n Id: ~p" + "~n UserMod: ~p" + "~n UserData: ~p" "~n DefaultAgentConfig: ~p", [Id, UserMod, UserData, DefaultAgentConfig]), case (catch verify_user_behaviour(UserMod)) of ok -> case verify_user_agent_config(DefaultAgentConfig) of - ok -> - Config = default_agent_config(DefaultAgentConfig), + {ok, DefAgentConf} -> + Config = default_agent_config(DefAgentConf), {ok, #user{id = Id, mod = UserMod, data = UserData, @@ -1783,10 +1916,15 @@ verify_user({Id, _, _, _}) -> {error, {bad_user_id, Id}}. verify_user_agent_config(Conf) -> - case verify_invalid(Conf, [user_id, engine_id, address]) of - ok -> - verify_agent_config2(Conf); - Error -> + try + begin + verify_invalid(Conf, [user_id, engine_id, address]), + verify_agent_config2(Conf) + end + catch + throw:Error -> + ?vdebug("verify_user_agent_config -> throw" + "~n Error: ~p", [Error]), Error end. @@ -2147,6 +2285,16 @@ handle_call({unregister_agent, UserId, TargetName}, _From, State) -> Reply = handle_unregister_agent(UserId, TargetName), {reply, Reply, State}; +handle_call({update_agent_info, UserId, TargetName, Info}, + _From, State) -> + ?vlog("received update_agent_info request: " + "~n UserId: ~p" + "~n TargetName: ~p" + "~n Info: ~p", [UserId, TargetName, Info]), + Reply = handle_update_agent_info(UserId, TargetName, Info), + {reply, Reply, State}; + +%% <BACKWARD-COMPAT> handle_call({update_agent_info, UserId, TargetName, Item, Val}, _From, State) -> ?vlog("received update_agent_info request: " @@ -2156,6 +2304,7 @@ handle_call({update_agent_info, UserId, TargetName, Item, Val}, "~n Val: ~p", [UserId, TargetName, Item, Val]), Reply = handle_update_agent_info(UserId, TargetName, Item, Val), {reply, Reply, State}; +%% </BACKWARD-COMPAT> handle_call({register_usm_user, User}, _From, State) -> ?vlog("received register_usm_user request: " @@ -2517,16 +2666,27 @@ handle_register_agent(UserId, TargetName, Config) -> "~n Config: ~p", [UserId, TargetName, Config]), case (catch agent_info(TargetName, user_id)) of {error, _} -> + ?vtrace("handle_register_agent -> user_id not found in config", []), case ets:lookup(snmpm_user_table, UserId) of [#user{default_agent_config = DefConfig}] -> + ?vtrace("handle_register_agent -> " + "~n DefConfig: ~p", [DefConfig]), + %% First, insert this users default config + ?vtrace("handle_register_agent -> store default config", []), do_handle_register_agent(TargetName, DefConfig), + %% Second, insert the config for this agent + ?vtrace("handle_register_agent -> store config", []), do_handle_register_agent(TargetName, [{user_id, UserId}|Config]), %% <DIRTY-BACKWARD-COMPATIBILLITY> %% And now for some (backward compatibillity) %% dirty crossref stuff + ?vtrace("handle_register_agent -> lookup address", []), {ok, Addr} = agent_info(TargetName, address), + ?vtrace("handle_register_agent -> Addr: ~p, lookup Port", + [Addr]), {ok, Port} = agent_info(TargetName, port), + ?vtrace("handle_register_agent -> register cross-ref fix", []), ets:insert(snmpm_agent_table, {{Addr, Port, target_name}, TargetName}), %% </DIRTY-BACKWARD-COMPATIBILLITY> @@ -2551,10 +2711,18 @@ handle_register_agent(UserId, TargetName, Config) -> do_handle_register_agent(_TargetName, []) -> ok; do_handle_register_agent(TargetName, [{Item, Val}|Rest]) -> + ?vtrace("handle_register_agent -> entry with" + "~n TargetName: ~p" + "~n Item: ~p" + "~n Val: ~p" + "~n Rest: ~p", [TargetName, Item, Val, Rest]), case (catch do_update_agent_info(TargetName, Item, Val)) of ok -> do_handle_register_agent(TargetName, Rest); {error, Reason} -> + ?vtrace("handle_register_agent -> failed updating ~p" + "~n Item: ~p" + "~n Reason: ~p", [Item, Reason]), ets:match_delete(snmpm_agent_table, {TargetName, '_'}), {error, Reason} end; @@ -2589,41 +2757,61 @@ handle_unregister_agent(UserId, TargetName) -> end. -handle_update_agent_info(UserId, TargetName, Item, Val) -> +handle_update_agent_info(UserId, TargetName, Info) -> ?vdebug("handle_update_agent_info -> entry with" "~n UserId: ~p" "~n TargetName: ~p" - "~n Item: ~p" - "~n Val: ~p", [UserId, TargetName, Item, Val]), + "~n Info: ~p", [UserId, TargetName, Info]), + %% Verify ownership case (catch agent_info(TargetName, user_id)) of - {ok, UserId} -> - do_update_agent_info(TargetName, Item, Val); + {ok, UserId} -> + handle_update_agent_info(TargetName, Info); {ok, OtherUserId} -> {error, {not_owner, OtherUserId}}; Error -> Error end. -do_update_agent_info(TargetName, Item, Val0) -> -%% p("do_update_agent_info -> entry with" -%% "~n TargetName: ~p" -%% "~n Item: ~p" -%% "~n Val0: ~p", [TargetName, Item, Val0]), - case verify_val(Item, Val0) of - {ok, Val} -> -%% p("do_update_agent_info -> verified value" -%% "~n Val: ~p", [Val]), - ets:insert(snmpm_agent_table, {{TargetName, Item}, Val}), - ok; +handle_update_agent_info(TargetName, Info0) -> + ?vtrace("handle_update_agent_info -> entry with" + "~n TargetName: ~p" + "~n Info0: ~p", [TargetName, Info0]), + %% Verify info + try verify_agent_info(TargetName, Info0) of + {ok, Info} -> + do_update_agent_info(TargetName, Info); Error -> - ?vlog("do_update_agent_info -> verify value failed: " - "~n TargetName: ~p" - "~n Item: ~p" - "~n Val0: ~p" - "~n Error: ~p", [TargetName, Item, Val0, Error]), - {error, {bad_agent_val, TargetName, Item, Val0}} + Error + catch + throw:Error -> + Error; + T:E -> + {error, {failed_info_verification, Info0, T, E}} end. +handle_update_agent_info(UserId, TargetName, Item, Val) -> + ?vdebug("handle_update_agent_info -> entry with" + "~n UserId: ~p" + "~n TargetName: ~p" + "~n Item: ~p" + "~n Val: ~p", [UserId, TargetName, Item, Val]), + handle_update_agent_info(TargetName, [{Item, Val}]). + +do_update_agent_info(TargetName, Info) -> + InsertItem = + fun({Item, Val}) -> + ets:insert(snmpm_agent_table, {{TargetName, Item}, Val}) + end, + lists:foreach(InsertItem, Info). + +do_update_agent_info(TargetName, Item, Val) -> + ?vtrace("do_update_agent_info -> entry with" + "~n TargetName: ~p" + "~n Item: ~p" + "~n Val: ~p", [TargetName, Item, Val]), + ets:insert(snmpm_agent_table, {{TargetName, Item}, Val}), + ok. + handle_register_usm_user(#usm_user{engine_id = EngineID, name = Name} = User) -> @@ -2791,7 +2979,7 @@ verify_mandatory(Conf, [Mand|Mands]) -> true -> verify_mandatory(Conf, Mands); false -> - {error, {missing_mandatory_config, Mand}} + throw({error, {missing_mandatory_config, Mand}}) end. verify_invalid(_, []) -> @@ -2801,7 +2989,7 @@ verify_invalid(Conf, [Inv|Invs]) -> false -> verify_invalid(Conf, Invs); true -> - {error, {illegal_config, Inv}} + throw({error, {illegal_config, Inv}}) end. @@ -2810,10 +2998,26 @@ verify_val(user_id, UserId) -> verify_val(reg_type, RegType) when (RegType =:= addr_port) orelse (RegType =:= target_name) -> {ok, RegType}; -verify_val(address, Addr0) -> - case normalize_address(Addr0) of +verify_val(tdomain = Item, snmpUDPDomain = _Domain) -> + verify_val(Item, transportDomainUdpIpv4); +verify_val(tdomain, Domain) -> + case lists:member(Domain, ?SUPPORTED_DOMAINS) of + true -> + {ok, Domain}; + false -> + case lists:member(Domain, snmp_conf:all_domains()) of + true -> + error({unsupported_domain, Domain}); + false -> + error({unknown_domain, Domain}) + end + end; +verify_val(address, {Domain, Addr0}) -> + case normalize_address(Domain, Addr0) of {_A1, _A2, _A3, _A4} = Addr -> {ok, Addr}; + {_A1, _A2, _A3, _A4, _A5, _A6, _A7, _A8} = Addr -> + {ok, Addr}; _ when is_list(Addr0) -> case (catch snmp_conf:check_ip(Addr0)) of ok -> @@ -2824,6 +3028,8 @@ verify_val(address, Addr0) -> _ -> error({bad_address, Addr0}) end; +verify_val(address, BadAddress) -> + error({bad_address, BadAddress}); verify_val(port, Port) -> case (catch snmp_conf:check_integer(Port, {gt, 0})) of ok -> @@ -2875,7 +3081,7 @@ verify_val(sec_name, BadName) -> verify_val(sec_level, Level) -> (catch snmp_conf:check_sec_level(Level)); verify_val(Item, _) -> - {error, {no_such_item, Item}}. + {error, {unknown_item, Item}}. %%%------------------------------------------------------------------- @@ -3034,11 +3240,17 @@ init_mini_mib_elems(MibName, [_|T], Res) -> %%---------------------------------------------------------------------- normalize_address(Addr) -> - case inet:getaddr(Addr, inet) of + normalize_address(snmpUDPDomain, Addr). + +normalize_address(snmpUDPDomain, Addr) -> + normalize_address(transportDomainUdpIpv4, Addr); + +normalize_address(Domain, Addr) -> + case inet:getaddr(Addr, td2fam(Domain)) of {ok, Addr2} -> Addr2; _ when is_list(Addr) -> - case (catch snmp_conf:check_ip(Addr)) of + case (catch snmp_conf:check_ip(Domain, Addr)) of ok -> list_to_tuple(Addr); _ -> @@ -3048,6 +3260,9 @@ normalize_address(Addr) -> Addr end. +td2fam(transportDomainUdpIpv4) -> inet; +td2fam(transportDomainUdpIpv6) -> inet6. + %%---------------------------------------------------------------------- diff --git a/lib/snmp/src/manager/snmpm_mpd.erl b/lib/snmp/src/manager/snmpm_mpd.erl index 7712370d28..627838e3d4 100644 --- a/lib/snmp/src/manager/snmpm_mpd.erl +++ b/lib/snmp/src/manager/snmpm_mpd.erl @@ -92,7 +92,7 @@ reset(#state{v3 = V3}) -> %% Purpose: This is the main Message Dispatching function. (see %% section 4.2.1 in rfc2272) %%----------------------------------------------------------------- -process_msg(Msg, TDomain, Addr, Port, State, NoteStore, Logger) -> +process_msg(Msg, Domain, Addr, Port, State, NoteStore, Logger) -> inc(snmpInPkts), @@ -102,18 +102,18 @@ process_msg(Msg, TDomain, Addr, Port, State, NoteStore, Logger) -> #message{version = 'version-1', vsn_hdr = Community, data = Data} when State#state.v1 =:= true -> HS = ?empty_msg_size + length(Community), - process_v1_v2c_msg('version-1', NoteStore, Msg, TDomain, - Addr, Port, + process_v1_v2c_msg('version-1', NoteStore, Msg, + Domain, Addr, Port, Community, Data, HS, Logger); %% Version 2 #message{version = 'version-2', vsn_hdr = Community, data = Data} when State#state.v2c =:= true -> HS = ?empty_msg_size + length(Community), - process_v1_v2c_msg('version-2', NoteStore, Msg, TDomain, - Addr, Port, - Community, Data, HS, Logger); - + (catch process_v1_v2c_msg('version-2', NoteStore, Msg, + Domain, Addr, Port, + Community, Data, HS, Logger)); + %% Version 3 #message{version = 'version-3', vsn_hdr = H, data = Data} when State#state.v3 =:= true -> @@ -148,17 +148,30 @@ process_msg(Msg, TDomain, Addr, Port, State, NoteStore, Logger) -> %%----------------------------------------------------------------- %% Handles a Community based message (v1 or v2c). %%----------------------------------------------------------------- -process_v1_v2c_msg(Vsn, _NoteStore, Msg, snmpUDPDomain, +process_v1_v2c_msg(Vsn, _NoteStore, Msg, Domain, Addr, Port, Community, Data, HS, Log) -> ?vdebug("process_v1_v2c_msg -> entry with" "~n Vsn: ~p" + "~n Domain: ~p" "~n Addr: ~p" "~n Port: ~p" "~n Community: ~p" - "~n HS: ~p", [Vsn, Addr, Port, Community, HS]), + "~n HS: ~p", [Vsn, Domain, Addr, Port, Community, HS]), + {TDomain, TAddress} = + try + begin + TD = snmp_conf:mk_tdomain(Domain), + TA = snmp_conf:mk_taddress(Domain, Addr, Port), + {TD, TA} + end + catch + throw:{error, TReason} -> + throw({discarded, {badarg, Domain, TReason}}) + end, + Max = get_max_message_size(), AgentMax = get_agent_max_message_size(Addr, Port), PduMS = pdu_ms(Max, AgentMax, HS), @@ -170,14 +183,14 @@ process_v1_v2c_msg(Vsn, _NoteStore, Msg, snmpUDPDomain, ?vtrace("process_v1_v2c_msg -> was a pdu", []), Log(Msg), inc_snmp_in(Pdu), - MsgData = {Community, sec_model(Vsn)}, + MsgData = {Community, sec_model(Vsn), TDomain, TAddress}, {ok, Vsn, Pdu, PduMS, MsgData}; Trap when is_record(Trap, trappdu) -> ?vtrace("process_v1_v2c_msg -> was a trap", []), Log(Msg), inc_snmp_in(Trap), - MsgData = {Community, sec_model(Vsn)}, + MsgData = {Community, sec_model(Vsn), TDomain, TAddress}, {ok, Vsn, Trap, PduMS, MsgData}; {'EXIT', Reason} -> @@ -185,11 +198,7 @@ process_v1_v2c_msg(Vsn, _NoteStore, Msg, snmpUDPDomain, "~n Reason: ~p", [Reason]), inc(snmpInASNParseErrs), {discarded, Reason} - end; -process_v1_v2c_msg(_Vsn, _NoteStore, _Msg, TDomain, - _Addr, _Port, - _Comm, _HS, _Data, _Log) -> - {discarded, {badarg, TDomain}}. + end. pdu_ms(MgrMMS, AgentMMS, HS) when AgentMMS < MgrMMS -> AgentMMS - HS; @@ -482,8 +491,8 @@ generate_msg('version-3', NoteStore, Pdu, generate_v3_msg(NoteStore, Pdu, SecModel, SecName, SecLevel, CtxEngineID, CtxName, TargetName, Log); -generate_msg(Vsn, _NoteStore, Pdu, {Community, _SecModel}, Log) -> - generate_v1_v2c_msg(Vsn, Pdu, Community, Log). +generate_msg(Vsn, _NoteStore, Pdu, {Comm, _SecModel}, Log) -> + generate_v1_v2c_msg(Vsn, Pdu, Comm, Log). generate_v3_msg(NoteStore, Pdu, @@ -627,6 +636,8 @@ generate_response_msg('version-3', Pdu, generate_v3_response_msg(Pdu, MsgID, SecModel, SecName, SecLevel, CtxEngineID, CtxName, SecData, Log); generate_response_msg(Vsn, Pdu, {Comm, _SecModel}, Log) -> + generate_v1_v2c_response_msg(Vsn, Pdu, Comm, Log); +generate_response_msg(Vsn, Pdu, {Comm, _SecModel, _TDomain, _TAddress}, Log) -> generate_v1_v2c_response_msg(Vsn, Pdu, Comm, Log). diff --git a/lib/snmp/src/manager/snmpm_net_if.erl b/lib/snmp/src/manager/snmpm_net_if.erl index a116c9f26b..4d6bd9aa33 100644 --- a/lib/snmp/src/manager/snmpm_net_if.erl +++ b/lib/snmp/src/manager/snmpm_net_if.erl @@ -28,7 +28,8 @@ start_link/2, stop/1, send_pdu/6, % Backward compatibillity - send_pdu/7, + send_pdu/7, % Backward compatibillity + send_pdu/8, inform_response/4, @@ -101,16 +102,21 @@ stop(Pid) -> send_pdu(Pid, Pdu, Vsn, MsgData, Addr, Port) -> send_pdu(Pid, Pdu, Vsn, MsgData, Addr, Port, ?DEFAULT_EXTRA_INFO). -send_pdu(Pid, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo) +send_pdu(Pid, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo) -> + Domain = snmpm_config:default_transport_domain(), + send_pdu(Pid, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo). + +send_pdu(Pid, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo) when is_record(Pdu, pdu) -> ?d("send_pdu -> entry with" "~n Pid: ~p" "~n Pdu: ~p" "~n Vsn: ~p" "~n MsgData: ~p" + "~n Domain: ~p" "~n Addr: ~p" - "~n Port: ~p", [Pid, Pdu, Vsn, MsgData, Addr, Port]), - cast(Pid, {send_pdu, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo}). + "~n Port: ~p", [Pid, Pdu, Vsn, MsgData, Domain, Addr, Port]), + cast(Pid, {send_pdu, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo}). note_store(Pid, NoteStore) -> call(Pid, {note_store, NoteStore}). @@ -380,15 +386,17 @@ handle_call(Req, From, State) -> %% {noreply, State, Timeout} | %% {stop, Reason, State} (terminate/2 is called) %%-------------------------------------------------------------------- -handle_cast({send_pdu, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo}, State) -> +handle_cast({send_pdu, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo}, + State) -> ?vlog("received send_pdu message with" "~n Pdu: ~p" "~n Vsn: ~p" "~n MsgData: ~p" + "~n Domain: ~p" "~n Addr: ~p" - "~n Port: ~p", [Pdu, Vsn, MsgData, Addr, Port]), + "~n Port: ~p", [Pdu, Vsn, MsgData, Domain, Addr, Port]), maybe_process_extra_info(ExtraInfo), - maybe_handle_send_pdu(Pdu, Vsn, MsgData, Addr, Port, State), + maybe_handle_send_pdu(Pdu, Vsn, MsgData, Domain, Addr, Port, State), {noreply, State}; handle_cast({inform_response, Ref, Addr, Port}, State) -> @@ -545,8 +553,9 @@ handle_recv_msg(Addr, Port, Bytes, mpd_state = MpdState, sock = Sock, log = Log} = State) -> + Domain = snmp_conf:which_domain(Addr), % What the ****... Logger = logger(Log, read, Addr, Port), - case (catch snmpm_mpd:process_msg(Bytes, snmpUDPDomain, Addr, Port, + case (catch snmpm_mpd:process_msg(Bytes, Domain, Addr, Port, MpdState, NoteStore, Logger)) of {ok, Vsn, Pdu, MS, ACM} -> @@ -734,17 +743,17 @@ irgc_stop(Ref) -> (catch erlang:cancel_timer(Ref)). -maybe_handle_send_pdu(Pdu, Vsn, MsgData, Addr, Port, +maybe_handle_send_pdu(Pdu, Vsn, MsgData, Domain, Addr, Port, #state{filter = FilterMod} = State) -> case (catch FilterMod:accept_send_pdu(Addr, Port, pdu_type_of(Pdu))) of false -> inc(netIfPduOutDrops), ok; _ -> - handle_send_pdu(Pdu, Vsn, MsgData, Addr, Port, State) + handle_send_pdu(Pdu, Vsn, MsgData, Domain, Addr, Port, State) end. -handle_send_pdu(Pdu, Vsn, MsgData, Addr, Port, +handle_send_pdu(Pdu, Vsn, MsgData, _Domain, Addr, Port, #state{server = Pid, note_store = NoteStore, sock = Sock, diff --git a/lib/snmp/src/manager/snmpm_server.erl b/lib/snmp/src/manager/snmpm_server.erl index 58a58507d6..484954addb 100644 --- a/lib/snmp/src/manager/snmpm_server.erl +++ b/lib/snmp/src/manager/snmpm_server.erl @@ -161,7 +161,8 @@ {id, user_id, reg_type, - target, + target, + domain, addr, port, type, @@ -1175,11 +1176,12 @@ handle_sync_get(Pid, UserId, TargetName, Oids, SendOpts, From, State) -> "~n From: ~p", [Pid, UserId, TargetName, Oids, SendOpts, From]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_sync_get -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), ReqId = send_get_request(Oids, Vsn, MsgData, - Addr, Port, Extra, State), + Domain, Addr, Port, + Extra, State), ?vdebug("handle_sync_get -> ReqId: ~p", [ReqId]), Msg = {sync_timeout, ReqId, From}, Timeout = ?SYNC_GET_TIMEOUT(SendOpts), @@ -1190,6 +1192,7 @@ handle_sync_get(Pid, UserId, TargetName, Oids, SendOpts, From, State) -> user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = get, @@ -1227,11 +1230,12 @@ handle_sync_get_next(Pid, UserId, TargetName, Oids, SendOpts, "~n From: ~p", [Pid, UserId, TargetName, Oids, SendOpts, From]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_sync_get_next -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), ReqId = send_get_next_request(Oids, Vsn, MsgData, - Addr, Port, Extra, State), + Domain, Addr, Port, + Extra, State), ?vdebug("handle_sync_get_next -> ReqId: ~p", [ReqId]), Msg = {sync_timeout, ReqId, From}, Timeout = ?SYNC_GET_NEXT_TIMEOUT(SendOpts), @@ -1242,6 +1246,7 @@ handle_sync_get_next(Pid, UserId, TargetName, Oids, SendOpts, user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = get_next, @@ -1285,10 +1290,11 @@ handle_sync_get_bulk(Pid, UserId, TargetName, NonRep, MaxRep, Oids, SendOpts, "~n From: ~p", [Pid, UserId, TargetName, NonRep, MaxRep, Oids, SendOpts, From]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_sync_get_bulk -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), - ReqId = send_get_bulk_request(Oids, Vsn, MsgData, Addr, Port, + ReqId = send_get_bulk_request(Oids, Vsn, MsgData, + Domain, Addr, Port, NonRep, MaxRep, Extra, State), ?vdebug("handle_sync_get_bulk -> ReqId: ~p", [ReqId]), Msg = {sync_timeout, ReqId, From}, @@ -1300,6 +1306,7 @@ handle_sync_get_bulk(Pid, UserId, TargetName, NonRep, MaxRep, Oids, SendOpts, user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = get_bulk, @@ -1339,11 +1346,12 @@ handle_sync_set(Pid, UserId, TargetName, VarsAndVals, SendOpts, From, State) -> "~n From: ~p", [Pid, UserId, TargetName, VarsAndVals, From]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_sync_set -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), ReqId = send_set_request(VarsAndVals, Vsn, MsgData, - Addr, Port, Extra, State), + Domain, Addr, Port, + Extra, State), ?vdebug("handle_sync_set -> ReqId: ~p", [ReqId]), Msg = {sync_timeout, ReqId, From}, Timeout = ?SYNC_SET_TIMEOUT(SendOpts), @@ -1354,6 +1362,7 @@ handle_sync_set(Pid, UserId, TargetName, VarsAndVals, SendOpts, From, State) -> user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = set, @@ -1391,10 +1400,11 @@ handle_async_get(Pid, UserId, TargetName, Oids, SendOpts, State) -> "~n SendOpts: ~p", [Pid, UserId, TargetName, Oids, SendOpts]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_async_get -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), - ReqId = send_get_request(Oids, Vsn, MsgData, Addr, Port, + ReqId = send_get_request(Oids, Vsn, MsgData, + Domain, Addr, Port, Extra, State), ?vdebug("handle_async_get -> ReqId: ~p", [ReqId]), Expire = ?ASYNC_GET_TIMEOUT(SendOpts), @@ -1402,6 +1412,7 @@ handle_async_get(Pid, UserId, TargetName, Oids, SendOpts, State) -> user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = get, @@ -1439,17 +1450,19 @@ handle_async_get_next(Pid, UserId, TargetName, Oids, SendOpts, State) -> "~n SendOpts: ~p", [Pid, UserId, TargetName, Oids, SendOpts]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_async_get_next -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), ReqId = send_get_next_request(Oids, Vsn, MsgData, - Addr, Port, Extra, State), + Domain, Addr, Port, + Extra, State), ?vdebug("handle_async_get_next -> ReqId: ~p", [ReqId]), Expire = ?ASYNC_GET_NEXT_TIMEOUT(SendOpts), Req = #request{id = ReqId, user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = get_next, @@ -1494,10 +1507,11 @@ handle_async_get_bulk(Pid, "~n SendOpts: ~p", [Pid, UserId, TargetName, NonRep, MaxRep, Oids, SendOpts]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_async_get_bulk -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), - ReqId = send_get_bulk_request(Oids, Vsn, MsgData, Addr, Port, + ReqId = send_get_bulk_request(Oids, Vsn, MsgData, + Domain, Addr, Port, NonRep, MaxRep, Extra, State), ?vdebug("handle_async_get_bulk -> ReqId: ~p", [ReqId]), Expire = ?ASYNC_GET_BULK_TIMEOUT(SendOpts), @@ -1505,6 +1519,7 @@ handle_async_get_bulk(Pid, user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = get_bulk, @@ -1541,17 +1556,19 @@ handle_async_set(Pid, UserId, TargetName, VarsAndVals, SendOpts, State) -> "~n SendOpts: ~p", [Pid, UserId, TargetName, VarsAndVals, SendOpts]), case agent_data(TargetName, SendOpts) of - {ok, RegType, Addr, Port, Vsn, MsgData} -> + {ok, RegType, Domain, Addr, Port, Vsn, MsgData} -> ?vtrace("handle_async_set -> send a ~p message", [Vsn]), Extra = ?GET_EXTRA(SendOpts), ReqId = send_set_request(VarsAndVals, Vsn, MsgData, - Addr, Port, Extra, State), + Domain, Addr, Port, + Extra, State), ?vdebug("handle_async_set -> ReqId: ~p", [ReqId]), Expire = ?ASYNC_SET_TIMEOUT(SendOpts), Req = #request{id = ReqId, user_id = UserId, reg_type = RegType, target = TargetName, + domain = Domain, addr = Addr, port = Port, type = set, @@ -2907,7 +2924,7 @@ do_gc(Key, Now) -> %% %%---------------------------------------------------------------------- -send_get_request(Oids, Vsn, MsgData, Addr, Port, ExtraInfo, +send_get_request(Oids, Vsn, MsgData, Domain, Addr, Port, ExtraInfo, #state{net_if = NetIf, net_if_mod = Mod, mini_mib = MiniMIB}) -> @@ -2918,34 +2935,39 @@ send_get_request(Oids, Vsn, MsgData, Addr, Port, ExtraInfo, "~n Pdu: ~p" "~n Vsn: ~p" "~n MsgData: ~p" + "~n Domain: ~p" "~n Addr: ~p" - "~n Port: ~p", [Mod, NetIf, Pdu, Vsn, MsgData, Addr, Port]), - (catch Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo)), + "~n Port: ~p", + [Mod, NetIf, Pdu, Vsn, MsgData, Domain, Addr, Port]), + Res = (catch Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, + Domain, Addr, Port, ExtraInfo)), + ?vtrace("send_get_request -> send result:" + "~n ~p", [Res]), Pdu#pdu.request_id. -send_get_next_request(Oids, Vsn, MsgData, Addr, Port, ExtraInfo, +send_get_next_request(Oids, Vsn, MsgData, Domain, Addr, Port, ExtraInfo, #state{mini_mib = MiniMIB, net_if = NetIf, net_if_mod = Mod}) -> Pdu = make_pdu(get_next, Oids, MiniMIB), - Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo), + Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo), Pdu#pdu.request_id. -send_get_bulk_request(Oids, Vsn, MsgData, Addr, Port, +send_get_bulk_request(Oids, Vsn, MsgData, Domain, Addr, Port, NonRep, MaxRep, ExtraInfo, #state{mini_mib = MiniMIB, net_if = NetIf, net_if_mod = Mod}) -> Pdu = make_pdu(bulk, {NonRep, MaxRep, Oids}, MiniMIB), - Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo), + Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo), Pdu#pdu.request_id. -send_set_request(VarsAndVals, Vsn, MsgData, Addr, Port, ExtraInfo, +send_set_request(VarsAndVals, Vsn, MsgData, Domain, Addr, Port, ExtraInfo, #state{mini_mib = MiniMIB, net_if = NetIf, net_if_mod = Mod}) -> Pdu = make_pdu(set, VarsAndVals, MiniMIB), - Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Addr, Port, ExtraInfo), + Mod:send_pdu(NetIf, Pdu, Vsn, MsgData, Domain, Addr, Port, ExtraInfo), Pdu#pdu.request_id. %% send_discovery(Vsn, MsgData, Addr, Port, ExtraInfo, @@ -3181,10 +3203,11 @@ agent_data(TargetName, SendOpts) -> {Comm, SecModel} end, + Domain = agent_data_item(tdomain, Info), Addr = agent_data_item(address, Info), Port = agent_data_item(port, Info), RegType = agent_data_item(reg_type, Info), - {ok, RegType, Addr, Port, version(Version), MsgData}; + {ok, RegType, Domain, Addr, Port, version(Version), MsgData}; Error -> Error end. diff --git a/lib/snmp/src/misc/snmp_conf.erl b/lib/snmp/src/misc/snmp_conf.erl index 20f4455d10..7249def24e 100644 --- a/lib/snmp/src/misc/snmp_conf.erl +++ b/lib/snmp/src/misc/snmp_conf.erl @@ -37,7 +37,9 @@ check_timer/1, + all_domains/0, check_domain/1, + all_tdomains/0, check_tdomain/1, mk_tdomain/1, which_domain/1, @@ -345,6 +347,25 @@ check_sec_level(BadSecLevel) -> %% --------- +all_tdomains() -> + [ + ?transportDomainUdpIpv4, + ?transportDomainUdpIpv6, + ?transportDomainUdpIpv4z, + ?transportDomainUdpIpv6z, + ?transportDomainTcpIpv4, + ?transportDomainTcpIpv6, + ?transportDomainTcpIpv4z, + ?transportDomainTcpIpv6z, + ?transportDomainSctpIpv4, + ?transportDomainSctpIpv6, + ?transportDomainSctpIpv4z, + ?transportDomainSctpIpv6z, + ?transportDomainLocal, + ?transportDomainUdpDns, + ?transportDomainTcpDns, + ?transportDomainSctpDns + ]. check_tdomain(TDomain) -> SupportedTDomains = @@ -353,25 +374,7 @@ check_tdomain(TDomain) -> ?transportDomainUdpIpv4, ?transportDomainUdpIpv6 ], - AllTDomains = - [ - ?transportDomainUdpIpv4, - ?transportDomainUdpIpv6, - ?transportDomainUdpIpv4z, - ?transportDomainUdpIpv6z, - ?transportDomainTcpIpv4, - ?transportDomainTcpIpv6, - ?transportDomainTcpIpv4z, - ?transportDomainTcpIpv6z, - ?transportDomainSctpIpv4, - ?transportDomainSctpIpv6, - ?transportDomainSctpIpv4z, - ?transportDomainSctpIpv6z, - ?transportDomainLocal, - ?transportDomainUdpDns, - ?transportDomainTcpDns, - ?transportDomainSctpDns - ], + AllTDomains = all_tdomains(), case lists:member(TDomain, SupportedTDomains) of true -> ok; @@ -388,7 +391,7 @@ check_tdomain(TDomain) -> %% --------- mk_tdomain(snmpUDPDomain) -> - ?snmpUDPDomain; + mk_tdomain(transportDomainUdpIpv4); mk_tdomain(transportDomainUdpIpv4) -> ?transportDomainUdpIpv4; mk_tdomain(transportDomainUdpIpv6) -> @@ -474,6 +477,26 @@ do_check_timer(WaitFor, Factor, Incr, Retry) -> %% --------- +all_domains() -> + [ + transportDomainUdpIpv4, + transportDomainUdpIpv6, + transportDomainUdpIpv4z, + transportDomainUdpIpv6z, + transportDomainTcpIpv4, + transportDomainTcpIpv6, + transportDomainTcpIpv4z, + transportDomainTcpIpv6z, + transportDomainSctpIpv4, + transportDomainSctpIpv6, + transportDomainSctpIpv4z, + transportDomainSctpIpv6z, + transportDomainLocal, + transportDomainUdpDns, + transportDomainTcpDns, + transportDomainSctpDns + ]. + check_domain(Domain) -> SupportedDomains = [ @@ -481,25 +504,7 @@ check_domain(Domain) -> transportDomainUdpIpv4, transportDomainUdpIpv6 ], - AllDomains = - [ - transportDomainUdpIpv4, - transportDomainUdpIpv6, - transportDomainUdpIpv4z, - transportDomainUdpIpv6z, - transportDomainTcpIpv4, - transportDomainTcpIpv6, - transportDomainTcpIpv4z, - transportDomainTcpIpv6z, - transportDomainSctpIpv4, - transportDomainSctpIpv6, - transportDomainSctpIpv4z, - transportDomainSctpIpv6z, - transportDomainLocal, - transportDomainUdpDns, - transportDomainTcpDns, - transportDomainSctpDns - ], + AllDomains = all_domains(), case lists:member(Domain, SupportedDomains) of true -> ok; diff --git a/lib/snmp/src/misc/snmp_config.erl b/lib/snmp/src/misc/snmp_config.erl index fcbc6a88c9..6ab20e3e48 100644 --- a/lib/snmp/src/misc/snmp_config.erl +++ b/lib/snmp/src/misc/snmp_config.erl @@ -337,7 +337,7 @@ config_agent_sys() -> {dir, ATLDir}, {size, ATLSize}, {repair, ATLRepair}, - {seqno, ATLSeqNo}]}]; + {seqno, ATLSeqNo}]}]; no -> [] end, @@ -568,7 +568,7 @@ config_agent_snmp(Dir, Vsns) -> false -> ok end, - i("The following agent files were written: agent.conf, " + i("The following agent files where written: agent.conf, " "community.conf,~n" "standard.conf, target_addr.conf, " "target_params.conf, ~n" @@ -776,7 +776,7 @@ config_manager_snmp(Dir, Vsns) -> Users, Agents, Usms)) of ok -> i("~n- - - - - - - - - - - - -"), - i("The following manager files were written: " + i("The following manager files where written: " "manager.conf, agents.conf " ++ case lists:member(v3, Vsns) of true -> @@ -2350,7 +2350,9 @@ write_sys_config_file_manager_atl_opt(Fid, {type, Type}) -> write_sys_config_file_manager_atl_opt(Fid, {size, Size}) -> ok = io:format(Fid, "{size, ~w}", [Size]); write_sys_config_file_manager_atl_opt(Fid, {repair, Rep}) -> - ok = io:format(Fid, "{repair, ~w}", [Rep]). + ok = io:format(Fid, "{repair, ~w}", [Rep]); +write_sys_config_file_manager_atl_opt(Fid, {seqno, SeqNo}) -> + ok = io:format(Fid, "{seqno, ~w}", [SeqNo]). header() -> diff --git a/lib/snmp/test/snmp_compiler_test.erl b/lib/snmp/test/snmp_compiler_test.erl index 2e6020ae7a..cee11ba97a 100644 --- a/lib/snmp/test/snmp_compiler_test.erl +++ b/lib/snmp/test/snmp_compiler_test.erl @@ -47,6 +47,7 @@ module_identity/1, agent_capabilities/1, module_compliance/1, + warnings_as_errors/1, otp_6150/1, otp_8574/1, @@ -97,9 +98,10 @@ all() -> description, oid_conflicts, imports, - module_identity, - agent_capabilities, - module_compliance, + module_identity, + agent_capabilities, + module_compliance, + warnings_as_errors, {group, tickets} ]. @@ -152,6 +154,8 @@ description(Config) when is_list(Config) -> ok. +%%====================================================================== + oid_conflicts(suite) -> []; oid_conflicts(Config) when is_list(Config) -> put(tname,oid_conflicts), @@ -165,18 +169,24 @@ oid_conflicts(Config) when is_list(Config) -> ok. +%%====================================================================== + imports(suite) -> []; imports(Config) when is_list(Config) -> ?SKIP(not_yet_implemented). +%%====================================================================== + module_identity(suite) -> []; module_identity(Config) when is_list(Config) -> ?SKIP(not_yet_implemented). +%%====================================================================== + agent_capabilities(suite) -> []; agent_capabilities(Config) when is_list(Config) -> @@ -218,6 +228,8 @@ agent_capabilities(Config) when is_list(Config) -> ok. +%%====================================================================== + module_compliance(suite) -> []; module_compliance(Config) when is_list(Config) -> @@ -259,6 +271,31 @@ module_compliance(Config) when is_list(Config) -> ok. +%%====================================================================== + +warnings_as_errors(suite) -> + ["OTP-9437"]; +warnings_as_errors(Config) when is_list(Config) -> + put(tname,warnings_as_errors), + p("starting with Config: ~p~n", [Config]), + Dir = ?config(comp_dir, Config), + MibDir = ?config(mib_dir, Config), + MibFile = join(MibDir, "OTP8574-MIB.mib"), + OutFile = join(Dir, "OTP8574-MIB.bin"), + Opts = [{group_check, false}, + {outdir, Dir}, + {verbosity, trace}, + relaxed_row_name_assign_check], + {error, compilation_failed} = + snmpc:compile(MibFile, [warnings_as_errors|Opts]), + false = filelib:is_regular(OutFile), + {ok, _} = snmpc:compile(MibFile, Opts), + true = filelib:is_regular(OutFile), + ok. + + +%%====================================================================== + otp_6150(suite) -> []; otp_6150(Config) when is_list(Config) -> @@ -273,6 +310,8 @@ otp_6150(Config) when is_list(Config) -> ok. +%%====================================================================== + otp_8574(suite) -> []; otp_8574(Config) when is_list(Config) -> @@ -304,6 +343,8 @@ otp_8574(Config) when is_list(Config) -> end. +%%====================================================================== + otp_8595(suite) -> []; otp_8595(Config) when is_list(Config) -> diff --git a/lib/snmp/test/snmp_manager_test.erl b/lib/snmp/test/snmp_manager_test.erl index 0b536748fb..d18f20d359 100644 --- a/lib/snmp/test/snmp_manager_test.erl +++ b/lib/snmp/test/snmp_manager_test.erl @@ -61,6 +61,7 @@ register_agent1/1, register_agent2/1, + register_agent3/1, info/1, @@ -383,12 +384,12 @@ end_per_testcase2(Case, Config) -> all() -> [ {group, start_and_stop_tests}, - {group, misc_tests}, + {group, misc_tests}, {group, user_tests}, - {group, agent_tests}, + {group, agent_tests}, {group, request_tests}, {group, event_tests}, - discovery, + discovery, {group, tickets} ]. @@ -417,7 +418,8 @@ groups() -> {agent_tests, [], [ register_agent1, - register_agent2 + register_agent2, + register_agent3 ] }, {request_tests, [], @@ -477,14 +479,14 @@ groups() -> }, {event_tests, [], [ - trap1, - trap2, - inform1, - inform2, - inform3, - inform4, - inform_swarm, - report + trap1%% , + %% trap2, + %% inform1, + %% inform2, + %% inform3, + %% inform4, + %% inform_swarm, + %% report ] }, {tickets, [], @@ -1134,6 +1136,7 @@ register_agent1(suite) -> register_agent1(Config) when is_list(Config) -> process_flag(trap_exit, true), put(tname,ra1), + p("starting with Config: ~p~n", [Config]), ManagerNode = start_manager_node(), @@ -1164,7 +1167,7 @@ register_agent1(Config) when is_list(Config) -> p("manager info: ~p~n", [mgr_info(ManagerNode)]), - p("register user(s) calvin & hobbe"), + p("register user(s) user_alfa & user_beta"), ?line ok = mgr_register_user(ManagerNode, user_alfa, snmpm_user_default, []), ?line ok = mgr_register_user(ManagerNode, user_beta, snmpm_user_default, []), p("manager info: ~p~n", [mgr_info(ManagerNode)]), @@ -1293,7 +1296,7 @@ register_agent2(Config) when is_list(Config) -> p("manager info: ~p~n", [mgr_info(ManagerNode)]), - p("register user(s) calvin & hobbe"), + p("register user(s) user_alfa & user_beta"), ?line ok = mgr_register_user(ManagerNode, user_alfa, snmpm_user_default, []), ?line ok = mgr_register_user(ManagerNode, user_beta, snmpm_user_default, []), p("manager info: ~p~n", [mgr_info(ManagerNode)]), @@ -1348,7 +1351,7 @@ register_agent2(Config) when is_list(Config) -> end, p("manager info: ~p~n", [mgr_info(ManagerNode)]), - + p("unregister user user_alfa"), ?line ok = mgr_unregister_user(ManagerNode, user_alfa), @@ -1377,7 +1380,157 @@ register_agent2(Config) when is_list(Config) -> p("manager info: ~p~n", [mgr_info(ManagerNode)]), - p("unregister user hobbe"), + p("unregister user user_beta"), + ?line ok = mgr_unregister_user(ManagerNode, user_beta), + + p("manager info: ~p~n", [mgr_info(ManagerNode)]), + + ?SLEEP(1000), + + p("stop snmp application (with only manager)"), + ?line ok = stop_snmp(ManagerNode), + + ?SLEEP(1000), + + stop_node(ManagerNode), + + ?SLEEP(1000), + + p("end"), + ok. + + +%%====================================================================== + +register_agent3(doc) -> + ["Test registration of agents with the NEW interface functions " + "and specifying transport domain"]; +register_agent3(suite) -> + []; +register_agent3(Config) when is_list(Config) -> + process_flag(trap_exit, true), + put(tname, ra3), + p("starting with Config: ~p~n", [Config]), + + ManagerNode = start_manager_node(), + + ConfDir = ?config(manager_conf_dir, Config), + DbDir = ?config(manager_db_dir, Config), + LocalHost = snmp_test_lib:localhost(), + + + write_manager_conf(ConfDir), + + Opts = [{server, [{verbosity, trace}]}, + {net_if, [{verbosity, trace}]}, + {note_store, [{verbosity, trace}]}, + {config, [{verbosity, trace}, {dir, ConfDir}, {db_dir, DbDir}]}], + + + p("load snmp application"), + ?line ok = load_snmp(ManagerNode), + + p("set manager env for the snmp application"), + ?line ok = set_mgr_env(ManagerNode, Opts), + + p("starting snmp application (with only manager)"), + ?line ok = start_snmp(ManagerNode), + + p("started"), + + ?SLEEP(1000), + + p("manager info: ~p~n", [mgr_info(ManagerNode)]), + + p("register user(s) user_alfa & user_beta"), + ?line ok = mgr_register_user(ManagerNode, user_alfa, snmpm_user_default, []), + ?line ok = mgr_register_user(ManagerNode, user_beta, snmpm_user_default, []), + p("manager info: ~p~n", [mgr_info(ManagerNode)]), + + p("register agent(s)"), + TargetName1 = "agent2", + ?line ok = mgr_register_agent(ManagerNode, user_alfa, TargetName1, + [{tdomain, transportDomainUdpIpv4}, + {address, LocalHost}, + {port, 5001}, + {engine_id, "agentEngineId-1"}]), + TargetName2 = "agent3", + ?line ok = mgr_register_agent(ManagerNode, user_alfa, TargetName2, + [{tdomain, transportDomainUdpIpv6}, + {address, LocalHost}, + {port, 5002}, + {engine_id, "agentEngineId-2"}]), + TargetName3 = "agent4", + ?line {error, {unsupported_domain, _} = Reason4} = + mgr_register_agent(ManagerNode, user_beta, TargetName3, + [{tdomain, transportDomainTcpIpv4}, + {address, LocalHost}, + {port, 5003}, + {engine_id, "agentEngineId-3"}]), + p("Expected registration failure: ~p", [Reason4]), + TargetName4 = "agent5", + ?line {error, {unknown_domain, _} = Reason5} = + mgr_register_agent(ManagerNode, user_beta, TargetName4, + [{tdomain, transportDomainUdpIpv4_bad}, + {address, LocalHost}, + {port, 5004}, + {engine_id, "agentEngineId-4"}]), + p("Expected registration failure: ~p", [Reason5]), + + p("verify all agent(s): expect 2"), + case mgr_which_agents(ManagerNode) of + Agents1 when length(Agents1) =:= 2 -> + p("all agents: ~p~n", [Agents1]), + ok; + Agents1 -> + ?FAIL({agent_registration_failure, Agents1}) + end, + + p("verify user_alfa agent(s)"), + case mgr_which_agents(ManagerNode, user_alfa) of + Agents2 when length(Agents2) =:= 2 -> + p("calvin agents: ~p~n", [Agents2]), + ok; + Agents2 -> + ?FAIL({agent_registration_failure, Agents2}) + end, + + p("verify user_beta agent(s)"), + case mgr_which_agents(ManagerNode, user_beta) of + Agents3 when length(Agents3) =:= 0 -> + p("hobbe agents: ~p~n", [Agents3]), + ok; + Agents3 -> + ?FAIL({agent_registration_failure, Agents3}) + end, + + p("manager info: ~p~n", [mgr_info(ManagerNode)]), + + p("unregister user user_alfa"), + ?line ok = mgr_unregister_user(ManagerNode, user_alfa), + + p("verify all agent(s): expect 0"), + case mgr_which_agents(ManagerNode) of + Agents4 when length(Agents4) =:= 0 -> + p("all agents: ~p~n", [Agents4]), + ok; + Agents4 -> + ?FAIL({agent_unregistration_failure, Agents4}) + end, + p("manager info: ~p~n", [mgr_info(ManagerNode)]), + + p("verify all agent(s): expect 0"), + case mgr_which_agents(ManagerNode) of + [] -> + ok; + Agents5 -> + p("all agents: ~p~n", [Agents5]), + ?FAIL({agent_unregistration_failure, Agents5}) + end, + + p("manager info: ~p~n", [mgr_info(ManagerNode)]), + + p("unregister user user_beta"), ?line ok = mgr_unregister_user(ManagerNode, user_beta), p("manager info: ~p~n", [mgr_info(ManagerNode)]), diff --git a/lib/snmp/vsn.mk b/lib/snmp/vsn.mk index 29228fc59b..08251ab9ea 100644 --- a/lib/snmp/vsn.mk +++ b/lib/snmp/vsn.mk @@ -17,6 +17,6 @@ # # %CopyrightEnd% -SNMP_VSN = 4.20 +SNMP_VSN = 4.21 PRE_VSN = APP_VSN = "snmp-$(SNMP_VSN)$(PRE_VSN)" diff --git a/lib/ssh/doc/src/notes.xml b/lib/ssh/doc/src/notes.xml index 71f3941577..6fc4fdc43d 100644 --- a/lib/ssh/doc/src/notes.xml +++ b/lib/ssh/doc/src/notes.xml @@ -29,6 +29,20 @@ <file>notes.xml</file> </header> +<section><title>Ssh 2.0.8</title> + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Calling ssh_sftp:stop_channel/1 resulted in that the trap_exit flag was + set to true for the invoking process.</p> + <p> + Own Id: OTP-9386 Aux Id: seq11865</p> + </item> + </list> + </section> +</section> + <section><title>Ssh 2.0.7</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/ssh/src/ssh.appup.src b/lib/ssh/src/ssh.appup.src index 974145836c..150b7d86dd 100644 --- a/lib/ssh/src/ssh.appup.src +++ b/lib/ssh/src/ssh.appup.src @@ -19,13 +19,19 @@ {"%VSN%", [ - {"2.0.6", [{load_module, ssh_userreg, soft_purge, soft_purge, []}]}, + {"2.0.7", [{load_module, ssh_sftp, soft_purge, soft_purge, []}]}, + {"2.0.6", [{load_module, ssh_userreg, soft_purge, soft_purge, []}, + {load_module, ssh_sftp, soft_purge, soft_purge, []}]}, {"2.0.5", [{load_module, ssh_userreg, soft_purge, soft_purge, []}, + {load_module, ssh_sftp, soft_purge, soft_purge, []}, {load_module, ssh_connection_handler, soft_purge, soft_purge, [ssh_userreg]}]} ], [ - {"2.0.6", [{load_module, ssh_userreg, soft_purge, soft_purge, []}]}, + {"2.0.7", [{load_module, ssh_sftp, soft_purge, soft_purge, []}]}, + {"2.0.6", [{load_module, ssh_userreg, soft_purge, soft_purge, []}, + {load_module, ssh_sftp, soft_purge, soft_purge, []}]}, {"2.0.5", [{load_module, ssh_userreg, soft_purge, soft_purge, []}, + {load_module, ssh_sftp, soft_purge, soft_purge, []}, {load_module, ssh_connection_handler, soft_purge, soft_purge, [ssh_userreg]}]} ] }. diff --git a/lib/ssh/src/ssh_sftp.erl b/lib/ssh/src/ssh_sftp.erl index 59e09fdd0f..f000558100 100755 --- a/lib/ssh/src/ssh_sftp.erl +++ b/lib/ssh/src/ssh_sftp.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2005-2010. All Rights Reserved. +%% Copyright Ericsson AB 2005-2011. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in @@ -130,9 +130,9 @@ start_channel(Host, Port, Opts) -> end. stop_channel(Pid) -> - case process_info(Pid, [trap_exit]) of - [{trap_exit, Bool}] -> - process_flag(trap_exit, true), + case is_process_alive(Pid) of + true -> + OldValue = process_flag(trap_exit, true), link(Pid), exit(Pid, ssh_sftp_stop_channel), receive @@ -145,9 +145,9 @@ stop_channel(Pid) -> ok end end, - process_flag(trap_exit, Bool), + process_flag(trap_exit, OldValue), ok; - undefined -> + false -> ok end. diff --git a/lib/ssh/vsn.mk b/lib/ssh/vsn.mk index d79038df29..fe2b915d17 100644 --- a/lib/ssh/vsn.mk +++ b/lib/ssh/vsn.mk @@ -1,5 +1,5 @@ #-*-makefile-*- ; force emacs to enter makefile-mode -SSH_VSN = 2.0.7 +SSH_VSN = 2.0.8 APP_VSN = "ssh-$(SSH_VSN)" diff --git a/lib/ssl/c_src/esock_openssl.c b/lib/ssl/c_src/esock_openssl.c index 2621c9934e..0bc42958f0 100644 --- a/lib/ssl/c_src/esock_openssl.c +++ b/lib/ssl/c_src/esock_openssl.c @@ -1024,7 +1024,7 @@ static void info_callback(const SSL *ssl, int where, int ret) } } -/* This function is called whenever a SSL_CTX *ctx structure is +/* This function is called whenever an SSL_CTX *ctx structure is * freed. */ static void callback_data_free(void *parent, void *ptr, CRYPTO_EX_DATA *ad, diff --git a/lib/ssl/doc/src/notes.xml b/lib/ssl/doc/src/notes.xml index b2d17925fd..e090b4e1ef 100644 --- a/lib/ssl/doc/src/notes.xml +++ b/lib/ssl/doc/src/notes.xml @@ -554,7 +554,7 @@ Own Id: OTP-8224</p> </item> <item> - <p>A ssl:ssl_accept/3 could crash a connection if the + <p>An ssl:ssl_accept/3 could crash a connection if the timing was wrong.</p> <p>Removed info message if the socket closed without a proper disconnect from the ssl layer. </p> <p>ssl:send/2 is now blocking until the @@ -770,7 +770,7 @@ <item> <p> The new ssl implementation released as a alfa in this - version supports upgrading of a tcp connection to a ssl + version supports upgrading of a tcp connection to an ssl connection so that http client and servers may implement RFC 2817.</p> <p> @@ -789,7 +789,7 @@ very crippled as the control of the ssl-socket was deep down in openssl making it hard if not impossible to support all inet options, ipv6 and upgrade of a tcp - connection to a ssl connection. The alfa version has a + connection to an ssl connection. The alfa version has a few limitations that will be removed before the ssl-4.0 release. Main differences and limitations in the alfa are listed below.</p> diff --git a/lib/ssl/doc/src/ssl.xml b/lib/ssl/doc/src/ssl.xml index 566068beaf..0c4c8796be 100644 --- a/lib/ssl/doc/src/ssl.xml +++ b/lib/ssl/doc/src/ssl.xml @@ -35,7 +35,7 @@ <title>SSL</title> <list type="bulleted"> - <item>ssl requires the crypto an public_key applications.</item> + <item>ssl requires the crypto and public_key applications.</item> <item>Supported SSL/TLS-versions are SSL-3.0 and TLS-1.0 </item> <item>For security reasons sslv2 is not supported.</item> <item>Ephemeral Diffie-Hellman cipher suites are supported @@ -216,7 +216,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | application is encountered. Additionally it will be called when a certificate is considered valid by the path validation to allow access to each certificate in the path to the user - application. Note that the it will differentiate between the + application. Note that it will differentiate between the peer certificate and CA certificates by using valid_peer or valid as the second argument to the verify fun. See <seealso marker="public_key:cert_records">the public_key User's @@ -326,10 +326,10 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | </item> <tag>{fail_if_no_peer_cert, boolean()}</tag> - <item>Used together with {verify, verify_peer} by a ssl server. + <item>Used together with {verify, verify_peer} by an ssl server. If set to true, the server will fail if the client does not have a certificate to send, i.e. sends a empty certificate, if set to - false it will only fail if the client sends a invalid + false it will only fail if the client sends an invalid certificate (an empty certificate is considered valid). </item> @@ -343,10 +343,10 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | PeerCert, Compression, CipherSuite) -> boolean()}</tag> <item>Enables the ssl server to have a local policy for deciding if a session should be reused or not, - only meaning full if <c>reuse_sessions</c> is set to true. + only meaningful if <c>reuse_sessions</c> is set to true. SuggestedSessionId is a binary(), PeerCert is a DER encoded certificate, Compression is an enumeration integer - and CipherSuite of type ciphersuite(). + and CipherSuite is of type ciphersuite(). </item> </taglist> @@ -355,7 +355,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | <section> <title>General</title> - <p>When a ssl socket is in active mode (the default), data from the + <p>When an ssl socket is in active mode (the default), data from the socket is delivered to the owner of the socket in the form of messages: </p> @@ -396,7 +396,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | <name>connect(Socket, SslOptions, Timeout) -> {ok, SslSocket} | {error, Reason}</name> <fsummary> Upgrades a gen_tcp, or - equivalent, connected socket to a ssl socket. </fsummary> + equivalent, connected socket to an ssl socket. </fsummary> <type> <v>Socket = socket()</v> <v>SslOptions = [ssloption()]</v> @@ -405,7 +405,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | <v>Reason = term()</v> </type> <desc> <p>Upgrades a gen_tcp, or equivalent, - connected socket to a ssl socket i.e. performs the + connected socket to an ssl socket i.e. performs the client-side ssl handshake.</p> </desc> </func> @@ -428,12 +428,12 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | <func> <name>close(SslSocket) -> ok | {error, Reason}</name> - <fsummary>Close a ssl connection</fsummary> + <fsummary>Close an ssl connection</fsummary> <type> <v>SslSocket = sslsocket()</v> <v>Reason = term()</v> </type> - <desc><p>Close a ssl connection.</p> + <desc><p>Close an ssl connection.</p> </desc> </func> @@ -450,7 +450,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | <v>Reason = term()</v> </type> <desc><p>Assigns a new controlling process to the ssl-socket. A - controlling process is the owner of a ssl-socket, and receives + controlling process is the owner of an ssl-socket, and receives all messages from the socket.</p> </desc> </func> @@ -496,14 +496,14 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | <func> <name>listen(Port, Options) -> {ok, ListenSocket} | {error, Reason}</name> - <fsummary>Creates a ssl listen socket.</fsummary> + <fsummary>Creates an ssl listen socket.</fsummary> <type> <v>Port = integer()</v> <v>Options = options()</v> <v>ListenSocket = sslsocket()</v> </type> <desc> - <p>Creates a ssl listen socket.</p> + <p>Creates an ssl listen socket.</p> </desc> </func> @@ -587,6 +587,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | the socket is closed.</p> </desc> </func> + <func> <name>setopts(Socket, Options) -> ok | {error, Reason}</name> <fsummary>Set socket options.</fsummary> @@ -646,7 +647,7 @@ fun(OtpCert :: #'OTPCertificate'{}, Event :: {bad_cert, Reason :: atom()} | </type> <desc> <p> Upgrades a gen_tcp, or - equivalent, socket to a ssl socket i.e. performs the + equivalent, socket to an ssl socket i.e. performs the ssl server-side handshake.</p> <p><warning>Note that the listen socket should be in {active, false} mode before telling the client that the server is ready to upgrade diff --git a/lib/ssl/doc/src/ssl_protocol.xml b/lib/ssl/doc/src/ssl_protocol.xml index 6936408881..ca5cc8bc7a 100644 --- a/lib/ssl/doc/src/ssl_protocol.xml +++ b/lib/ssl/doc/src/ssl_protocol.xml @@ -31,11 +31,11 @@ </p> <p>By default erlang ssl is run over the TCP/IP protocol even - though you could plug in an other reliable transport protocol + though you could plug in any other reliable transport protocol with the same API as gen_tcp.</p> <p>If a client and server wants to use an upgrade mechanism, such as - defined by RFC2817, to upgrade a regular TCP/IP connection to a ssl + defined by RFC2817, to upgrade a regular TCP/IP connection to an ssl connection the erlang ssl API supports this. This can be useful for things such as supporting HTTP and HTTPS on the same port and implementing virtual hosting. diff --git a/lib/ssl/doc/src/using_ssl.xml b/lib/ssl/doc/src/using_ssl.xml index 605290b6f9..ab837a156a 100644 --- a/lib/ssl/doc/src/using_ssl.xml +++ b/lib/ssl/doc/src/using_ssl.xml @@ -56,7 +56,7 @@ <code type="erl">1 server> ssl:start(). ok</code> - <p>Create a ssl listen socket</p> + <p>Create an ssl listen socket</p> <code type="erl">2 server> {ok, ListenSocket} = ssl:listen(9999, [{certfile, "cert.pem"}, {keyfile, "key.pem"},{reuseaddr, true}]). {ok,{sslsocket, [...]}}</code> @@ -90,7 +90,7 @@ ok</code> <section> <title>Upgrade example</title> - <note><p> To upgrade a TCP/IP connection to a ssl connection the + <note><p> To upgrade a TCP/IP connection to an ssl connection the client and server have to aggre to do so. Agreement may be accompliced by using a protocol such the one used by HTTP specified in RFC 2817.</p> </note> @@ -114,7 +114,7 @@ ok</code> <code type="erl">2 client> {ok, Socket} = gen_tcp:connect("localhost", 9999, [], infinity).</code> <p>Make sure active is set to false before trying - to upgrade a connection to a ssl connection, otherwhise + to upgrade a connection to an ssl connection, otherwhise ssl handshake messages may be deliverd to the wrong process.</p> <code type="erl">4 server> inet:setopts(Socket, [{active, false}]). ok</code> @@ -124,7 +124,7 @@ ok</code> {certfile, "cert.pem"}, {keyfile, "key.pem"}]). {ok,{sslsocket,[...]}}</code> - <p> Upgrade to a ssl connection. Note that the client and server + <p> Upgrade to an ssl connection. Note that the client and server must agree upon the upgrade and the server must call ssl:accept/2 before the client calls ssl:connect/3.</p> <code type="erl">3 client>{ok, SSLSocket} = ssl:connect(Socket, [{cacertfile, "cacerts.pem"}, diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index a0aedbbbee..46e4b98c98 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -104,7 +104,7 @@ stop() -> {ok, #sslsocket{}} | {error, reason()}. %% -%% Description: Connect to a ssl server. +%% Description: Connect to an ssl server. %%-------------------------------------------------------------------- connect(Socket, SslOptions) when is_port(Socket) -> connect(Socket, SslOptions, infinity). @@ -151,7 +151,7 @@ connect(Host, Port, Options0, Timeout) -> -spec listen(port_num(), [option()]) ->{ok, #sslsocket{}} | {error, reason()}. %% -%% Description: Creates a ssl listen socket. +%% Description: Creates an ssl listen socket. %%-------------------------------------------------------------------- listen(_Port, []) -> {error, enooptions}; @@ -177,7 +177,7 @@ listen(Port, Options0) -> -spec transport_accept(#sslsocket{}, timeout()) -> {ok, #sslsocket{}} | {error, reason()}. %% -%% Description: Performs transport accept on a ssl listen socket +%% Description: Performs transport accept on an ssl listen socket %%-------------------------------------------------------------------- transport_accept(ListenSocket) -> transport_accept(ListenSocket, infinity). @@ -218,7 +218,7 @@ transport_accept(#sslsocket{} = ListenSocket, Timeout) -> ok | {ok, #sslsocket{}} | {error, reason()}. -spec ssl_accept(port(), [option()], timeout()) -> {ok, #sslsocket{}} | {error, reason()}. %% -%% Description: Performs accept on a ssl listen socket. e.i. performs +%% Description: Performs accept on an ssl listen socket. e.i. performs %% ssl handshake. %%-------------------------------------------------------------------- ssl_accept(ListenSocket) -> @@ -252,7 +252,7 @@ ssl_accept(Socket, SslOptions, Timeout) when is_port(Socket) -> %%-------------------------------------------------------------------- -spec close(#sslsocket{}) -> term(). %% -%% Description: Close a ssl connection +%% Description: Close an ssl connection %%-------------------------------------------------------------------- close(#sslsocket{pid = {ListenSocket, #config{cb={CbMod,_, _, _}}}, fd = new_ssl}) -> CbMod:close(ListenSocket); diff --git a/lib/ssl/src/ssl_connection.erl b/lib/ssl/src/ssl_connection.erl index 21b021afb0..79570c520a 100644 --- a/lib/ssl/src/ssl_connection.erl +++ b/lib/ssl/src/ssl_connection.erl @@ -131,7 +131,7 @@ recv(Pid, Length, Timeout) -> pid(), tuple(), timeout()) -> {ok, #sslsocket{}} | {error, reason()}. %% -%% Description: Connect to a ssl server. +%% Description: Connect to an ssl server. %%-------------------------------------------------------------------- connect(Host, Port, Socket, Options, User, CbInfo, Timeout) -> try start_fsm(client, Host, Port, Socket, Options, User, CbInfo, @@ -145,7 +145,7 @@ connect(Host, Port, Socket, Options, User, CbInfo, Timeout) -> pid(), tuple(), timeout()) -> {ok, #sslsocket{}} | {error, reason()}. %% -%% Description: Performs accept on a ssl listen socket. e.i. performs +%% Description: Performs accept on an ssl listen socket. e.i. performs %% ssl handshake. %%-------------------------------------------------------------------- ssl_accept(Port, Socket, Opts, User, CbInfo, Timeout) -> @@ -185,7 +185,7 @@ socket_control(Socket, Pid, CbModule) -> %%-------------------------------------------------------------------- -spec close(pid()) -> ok | {error, reason()}. %% -%% Description: Close a ssl connection +%% Description: Close an ssl connection %%-------------------------------------------------------------------- close(ConnectionPid) -> case sync_send_all_state_event(ConnectionPid, close) of diff --git a/lib/ssl/src/ssl_record.erl b/lib/ssl/src/ssl_record.erl index 4c3c0b9c58..72091fdd5f 100644 --- a/lib/ssl/src/ssl_record.erl +++ b/lib/ssl/src/ssl_record.erl @@ -342,7 +342,7 @@ get_tls_records_aux(<<?BYTE(?CHANGE_CIPHER_SPEC),?BYTE(MajVer),?BYTE(MinVer), get_tls_records_aux(Rest, [#ssl_tls{type = ?CHANGE_CIPHER_SPEC, version = {MajVer, MinVer}, fragment = Data} | Acc]); -%% Matches a ssl v2 client hello message. +%% Matches an ssl v2 client hello message. %% The server must be able to receive such messages, from clients that %% are willing to use ssl v3 or higher, but have ssl v2 compatibility. get_tls_records_aux(<<1:1, Length0:15, Data0:Length0/binary, Rest/binary>>, diff --git a/lib/ssl/src/ssl_ssl2.erl b/lib/ssl/src/ssl_ssl2.erl index b1005b1acb..30a3a5fc98 100644 --- a/lib/ssl/src/ssl_ssl2.erl +++ b/lib/ssl/src/ssl_ssl2.erl @@ -20,7 +20,7 @@ %% %%---------------------------------------------------------------------- %% Purpose: Handles sslv2 hello as clients supporting sslv2 and higher -%% will send a sslv2 hello. +%% will send an sslv2 hello. %%---------------------------------------------------------------------- -module(ssl_ssl2). diff --git a/lib/stdlib/doc/src/gen_fsm.xml b/lib/stdlib/doc/src/gen_fsm.xml index d15383c621..4900a8f0ef 100644 --- a/lib/stdlib/doc/src/gen_fsm.xml +++ b/lib/stdlib/doc/src/gen_fsm.xml @@ -438,7 +438,7 @@ gen_fsm:sync_send_all_state_event -----> Module:handle_sync_event/4 <fsummary>Initialize process and internal state name and state data.</fsummary> <type> <v>Args = term()</v> - <v>Return = {ok,StateName,StateData} | {ok,StateName,StateData,Timeout}</v> + <v>Result = {ok,StateName,StateData} | {ok,StateName,StateData,Timeout}</v> <v> | {ok,StateName,StateData,hibernate}</v> <v> | {stop,Reason} | ignore</v> <v> StateName = atom()</v> diff --git a/lib/stdlib/doc/src/unicode_usage.xml b/lib/stdlib/doc/src/unicode_usage.xml index 416df1f02c..b48ad8c1f3 100644 --- a/lib/stdlib/doc/src/unicode_usage.xml +++ b/lib/stdlib/doc/src/unicode_usage.xml @@ -52,7 +52,7 @@ <tag>UCS-4</tag> <item>Basically the same as UTF-32, but without some Unicode semantics, defined by IEEE and has little use as a separate encoding standard. For all normal (and possibly abnormal) usages, UTF-32 and UCS-4 are interchangeable.</item> </taglist> -<p>Certain ranges of characters are left unused and certain ranges are even deemed invalid. The most notable invalid range is 16#D800 - 16#DFFF, as the UTF-16 encoding does not allow for encoding of these numbers. It can be speculated that the UTF-16 encoding standard was, from the beginning, expected to be able to hold all Unicode characters in one 16-bit entity, but then had to be extended, leaving a whole in the Unicode range to cope with backward compatibility.</p> +<p>Certain ranges of characters are left unused and certain ranges are even deemed invalid. The most notable invalid range is 16#D800 - 16#DFFF, as the UTF-16 encoding does not allow for encoding of these numbers. It can be speculated that the UTF-16 encoding standard was, from the beginning, expected to be able to hold all Unicode characters in one 16-bit entity, but then had to be extended, leaving a hole in the Unicode range to cope with backward compatibility.</p> <p>Additionally, the codepoint 16#FEFF is used for byte order marks (BOM's) and use of that character is not encouraged in other contexts than that. It actually is valid though, as the character "ZWNBS" (Zero Width Non Breaking Space). BOM's are used to identify encodings and byte order for programs where such parameters are not known in advance. Byte order marks are more seldom used than one could expect, put their use is becoming more widely spread as they provide the means for programs to make educated guesses about the Unicode format of a certain file.</p> </section> <section> diff --git a/lib/stdlib/src/io_lib_fread.erl b/lib/stdlib/src/io_lib_fread.erl index 52aa4d073c..ded1346097 100644 --- a/lib/stdlib/src/io_lib_fread.erl +++ b/lib/stdlib/src/io_lib_fread.erl @@ -24,6 +24,10 @@ -import(lists, [reverse/1,reverse/2]). +-define(is_whitespace(C), + ((C) =:= $\s orelse (C) =:= $\t + orelse (C) =:= $\r orelse (C) =:= $\n)). + %%----------------------------------------------------------------------- %% fread(Continuation, CharList, FormatString) @@ -106,31 +110,27 @@ fread_line(Format0, Line, N0, Results0, More, Newline) -> fread(Format, Line) -> fread(Format, Line, 0, []). -fread([$~|Format0], Line, N, Results) -> +fread([$~|Format0]=AllFormat, Line, N, Results) -> {Format,F,Sup,Unicode} = fread_field(Format0), - fread1(Format, F, Sup, Unicode, Line, N, Results, Format0); -fread([$\s|Format], Line, N, Results) -> - fread_skip_white(Format, Line, N, Results); -fread([$\t|Format], Line, N, Results) -> - fread_skip_white(Format, Line, N, Results); -fread([$\r|Format], Line, N, Results) -> - fread_skip_white(Format, Line, N, Results); -fread([$\n|Format], Line, N, Results) -> + fread1(Format, F, Sup, Unicode, Line, N, Results, AllFormat); +fread([C|Format], Line, N, Results) when ?is_whitespace(C) -> fread_skip_white(Format, Line, N, Results); fread([C|Format], [C|Line], N, Results) -> fread(Format, Line, N+1, Results); fread([_F|_Format], [_C|_Line], _N, _Results) -> fread_error(input); +fread([_|_]=Format, [], N, Results) -> + {more,Format,N,Results}; +fread([_|_], eof, 0, []) -> + %% This is at start of input so no error. + eof; +fread([_|_], eof, _N, _Results) -> + %% This is an error as there is no more input. + fread_error(input); fread([], Line, _N, Results) -> {ok,reverse(Results),Line}. -fread_skip_white(Format, [$\s|Line], N, Results) -> - fread_skip_white(Format, Line, N+1, Results); -fread_skip_white(Format, [$\t|Line], N, Results) -> - fread_skip_white(Format, Line, N+1, Results); -fread_skip_white(Format, [$\r|Line], N, Results) -> - fread_skip_white(Format, Line, N+1, Results); -fread_skip_white(Format, [$\n|Line], N, Results) -> +fread_skip_white(Format, [C|Line], N, Results) when ?is_whitespace(C) -> fread_skip_white(Format, Line, N+1, Results); fread_skip_white(Format, Line, N, Results) -> fread(Format, Line, N, Results). @@ -166,9 +166,9 @@ fread1([$l|Format], _F, Sup, _U, Line, N, Res, _AllFormat) -> fread(Format, Line, N, fread_result(Sup, N, Res)); fread1(_Format, _F, _Sup, _U, [], N, Res, AllFormat) -> %% Need more input here. - {more,[$~|AllFormat],N,Res}; -fread1(_Format, _F, _Sup, _U, eof, _N, [], _AllFormat) -> - %% This is at start of format string so no error. + {more,AllFormat,N,Res}; +fread1(_Format, _F, _Sup, _U, eof, 0, [], _AllFormat) -> + %% This is at start of input so no error. eof; fread1(_Format, _F, _Sup, _U, eof, _N, _Res, _AllFormat) -> %% This is an error as there is no more input. @@ -386,26 +386,16 @@ fread_string_cs(Line0, N0, true) -> %% fread_digits(Line, N, Base, Characters) %% Read segments of things, return "thing" characters in reverse order. -fread_skip_white([$\s|Line]) -> fread_skip_white(Line); -fread_skip_white([$\t|Line]) -> fread_skip_white(Line); -fread_skip_white([$\r|Line]) -> fread_skip_white(Line); -fread_skip_white([$\n|Line]) -> fread_skip_white(Line); +fread_skip_white([C|Line]) when ?is_whitespace(C) -> + fread_skip_white(Line); fread_skip_white(Line) -> Line. -fread_skip_white([$\s|Line], N) -> - fread_skip_white(Line, N+1); -fread_skip_white([$\t|Line], N) -> - fread_skip_white(Line, N+1); -fread_skip_white([$\r|Line], N) -> - fread_skip_white(Line, N+1); -fread_skip_white([$\n|Line], N) -> +fread_skip_white([C|Line], N) when ?is_whitespace(C) -> fread_skip_white(Line, N+1); fread_skip_white(Line, N) -> {Line,N}. -fread_skip_latin1_nonwhite([$\s|Line], N, Cs) -> {[$\s|Line],N,Cs}; -fread_skip_latin1_nonwhite([$\t|Line], N, Cs) -> {[$\t|Line],N,Cs}; -fread_skip_latin1_nonwhite([$\r|Line], N, Cs) -> {[$\r|Line],N,Cs}; -fread_skip_latin1_nonwhite([$\n|Line], N, Cs) -> {[$\n|Line],N,Cs}; +fread_skip_latin1_nonwhite([C|Line], N, Cs) when ?is_whitespace(C) -> + {[C|Line],N,Cs}; fread_skip_latin1_nonwhite([C|Line], N, []) when C > 255 -> {[C|Line],N,error}; fread_skip_latin1_nonwhite([C|Line], N, Cs) when C > 255 -> @@ -414,10 +404,8 @@ fread_skip_latin1_nonwhite([C|Line], N, Cs) -> fread_skip_latin1_nonwhite(Line, N+1, [C|Cs]); fread_skip_latin1_nonwhite([], N, Cs) -> {[],N,Cs}. -fread_skip_nonwhite([$\s|Line], N, Cs) -> {[$\s|Line],N,Cs}; -fread_skip_nonwhite([$\t|Line], N, Cs) -> {[$\t|Line],N,Cs}; -fread_skip_nonwhite([$\r|Line], N, Cs) -> {[$\r|Line],N,Cs}; -fread_skip_nonwhite([$\n|Line], N, Cs) -> {[$\n|Line],N,Cs}; +fread_skip_nonwhite([C|Line], N, Cs) when ?is_whitespace(C) -> + {[C|Line],N,Cs}; fread_skip_nonwhite([C|Line], N, Cs) -> fread_skip_nonwhite(Line, N+1, [C|Cs]); fread_skip_nonwhite([], N, Cs) -> {[],N,Cs}. diff --git a/lib/stdlib/src/timer.erl b/lib/stdlib/src/timer.erl index e3d6c905b6..689e42051f 100644 --- a/lib/stdlib/src/timer.erl +++ b/lib/stdlib/src/timer.erl @@ -199,7 +199,7 @@ tc(M, F, A) -> %% Calculate the time difference (in microseconds) of two %% erlang:now() timestamps, T2-T1. %% --spec now_diff(T1, T2) -> Tdiff when +-spec now_diff(T2, T1) -> Tdiff when T1 :: erlang:timestamp(), T2 :: erlang:timestamp(), Tdiff :: integer(). diff --git a/lib/stdlib/src/zip.erl b/lib/stdlib/src/zip.erl index 524d709431..c82c8159b6 100644 --- a/lib/stdlib/src/zip.erl +++ b/lib/stdlib/src/zip.erl @@ -223,7 +223,7 @@ openzip_open(F, Options) -> do_openzip_open(F, Options) -> Opts = get_openzip_options(Options), #openzip_opts{output = Output, open_opts = OpO, cwd = CWD} = Opts, - Input = get_zip_input(F), + Input = get_input(F), In0 = Input({open, F, OpO -- [write]}, []), {[#zip_comment{comment = C} | Files], In1} = get_central_dir(In0, fun raw_file_info_etc/5, Input), @@ -489,7 +489,7 @@ do_list_dir(F, Options) -> %% Print zip directory in short form -spec(t(Archive) -> ok when - Archive :: file:name() | binary | ZipHandle, + Archive :: file:name() | binary() | ZipHandle, ZipHandle :: pid()). t(F) when is_pid(F) -> zip_t(F); @@ -513,7 +513,7 @@ do_t(F, RawPrint) -> %% Print zip directory in long form (like ls -l) -spec(tt(Archive) -> ok when - Archive :: file:name() | binary | ZipHandle, + Archive :: file:name() | binary() | ZipHandle, ZipHandle :: pid()). tt(F) when is_pid(F) -> zip_tt(F); @@ -1174,7 +1174,7 @@ zip_get(Pid) when is_pid(Pid) -> zip_close(Pid) when is_pid(Pid) -> request(self(), Pid, close). --spec(zip_get(FileName, ZipHandle) -> {ok, [Result]} | {error, Reason} when +-spec(zip_get(FileName, ZipHandle) -> {ok, Result} | {error, Reason} when FileName :: file:name(), ZipHandle :: pid(), Result :: file:name() | {file:name(), binary()}, @@ -1183,7 +1183,7 @@ zip_close(Pid) when is_pid(Pid) -> zip_get(FileName, Pid) when is_pid(Pid) -> request(self(), Pid, {get, FileName}). --spec(zip_list_dir(ZipHandle) -> Result | {error, Reason} when +-spec(zip_list_dir(ZipHandle) -> {ok, Result} | {error, Reason} when Result :: [zip_comment() | zip_file()], ZipHandle :: pid(), Reason :: term()). diff --git a/lib/stdlib/test/ets_SUITE.erl b/lib/stdlib/test/ets_SUITE.erl index 9d348b5f1a..9341300f90 100644 --- a/lib/stdlib/test/ets_SUITE.erl +++ b/lib/stdlib/test/ets_SUITE.erl @@ -72,6 +72,7 @@ exit_many_many_tables_owner/1]). -export([write_concurrency/1, heir/1, give_away/1, setopts/1]). -export([bad_table/1, types/1]). +-export([otp_9423/1]). -export([init_per_testcase/2, end_per_testcase/2]). %% Convenience for manual testing @@ -143,7 +144,8 @@ all() -> otp_8166, exit_large_table_owner, exit_many_large_table_owner, exit_many_tables_owner, exit_many_many_tables_owner, write_concurrency, heir, - give_away, setopts, bad_table, types]. + give_away, setopts, bad_table, types, + otp_9423]. groups() -> [{new, [], @@ -5420,7 +5422,39 @@ types_do(Opts) -> ?line verify_etsmem(EtsMem). - +otp_9423(doc) -> ["vm-deadlock caused by race between ets:delete and others on write_concurrency table"]; +otp_9423(Config) when is_list(Config) -> + InitF = fun(_) -> {0,0} end, + ExecF = fun({S,F}) -> + receive + stop -> + io:format("~p got stop\n", [self()]), + [end_of_work | {"Succeded=",S,"Failed=",F}] + after 0 -> + %%io:format("~p (~p) doing lookup\n", [self(), {S,F}]), + try ets:lookup(otp_9423, key) of + [] -> {S+1,F} + catch + error:badarg -> {S,F+1} + end + end + end, + FiniF = fun(R) -> R end, + case run_workers(InitF, ExecF, FiniF, infinite, 1) of + Pids when is_list(Pids) -> + %%[P ! start || P <- Pids], + repeat(fun() -> ets:new(otp_9423, [named_table, public, {write_concurrency,true}]), + ets:delete(otp_9423) + end, 10000), + [P ! stop || P <- Pids], + wait_pids(Pids), + ok; + + Skipped -> Skipped + end. + + + % % Utility functions: @@ -5434,21 +5468,30 @@ add_lists([E1|T1], [E2|T2], Acc) -> add_lists(T1, T2, [E1+E2 | Acc]). run_workers(InitF,ExecF,FiniF,Laps) -> + run_workers(InitF,ExecF,FiniF,Laps, 0). +run_workers(InitF,ExecF,FiniF,Laps, Exclude) -> case erlang:system_info(smp_support) of true -> - run_workers_do(InitF,ExecF,FiniF,Laps); + run_workers_do(InitF,ExecF,FiniF,Laps, Exclude); false -> {skipped,"No smp support"} end. - + run_workers_do(InitF,ExecF,FiniF,Laps) -> - NumOfProcs = erlang:system_info(schedulers), + run_workers_do(InitF,ExecF,FiniF,Laps, 0). +run_workers_do(InitF,ExecF,FiniF,Laps, Exclude) -> + ?line NumOfProcs = case erlang:system_info(schedulers) of + N when (N > Exclude) -> N - Exclude + end, io:format("smp starting ~p workers\n",[NumOfProcs]), Seeds = [{ProcN,random:uniform(9999)} || ProcN <- lists:seq(1,NumOfProcs)], Parent = self(), Pids = [spawn_link(fun()-> worker(Seed,InitF,ExecF,FiniF,Laps,Parent,NumOfProcs) end) || Seed <- Seeds], - wait_pids(Pids). + case Laps of + infinite -> Pids; + _ -> wait_pids(Pids) + end. worker({ProcN,Seed}, InitF, ExecF, FiniF, Laps, Parent, NumOfProcs) -> io:format("smp worker ~p, seed=~p~n",[self(),Seed]), @@ -5463,6 +5506,8 @@ worker_loop(0, _, State) -> State; worker_loop(_, _, [end_of_work|State]) -> State; +worker_loop(infinite, ExecF, State) -> + worker_loop(infinite,ExecF,ExecF(State)); worker_loop(N, ExecF, State) -> worker_loop(N-1,ExecF,ExecF(State)). @@ -5517,20 +5562,21 @@ etsmem() -> case erlang:system_info({allocator,ets_alloc}) of false -> undefined; MemInfo -> - MSBCS = lists:foldl( - fun ({instance, _, L}, Acc) -> - {value,{_,MBCS}} = lists:keysearch(mbcs, 1, L), - {value,{_,SBCS}} = lists:keysearch(sbcs, 1, L), - [MBCS,SBCS | Acc] - end, - [], - MemInfo), + CS = lists:foldl( + fun ({instance, _, L}, Acc) -> + {value,{_,SBMBCS}} = lists:keysearch(sbmbcs, 1, L), + {value,{_,MBCS}} = lists:keysearch(mbcs, 1, L), + {value,{_,SBCS}} = lists:keysearch(sbcs, 1, L), + [SBMBCS,MBCS,SBCS | Acc] + end, + [], + MemInfo), lists:foldl( fun(L, {Bl0,BlSz0}) -> {value,{_,Bl,_,_}} = lists:keysearch(blocks, 1, L), {value,{_,BlSz,_,_}} = lists:keysearch(blocks_size, 1, L), {Bl0+Bl,BlSz0+BlSz} - end, {0,0}, MSBCS) + end, {0,0}, CS) end}, {Mem,AllTabs}. diff --git a/lib/stdlib/test/io_SUITE.erl b/lib/stdlib/test/io_SUITE.erl index 54a98985cd..bb02a879c2 100644 --- a/lib/stdlib/test/io_SUITE.erl +++ b/lib/stdlib/test/io_SUITE.erl @@ -27,7 +27,7 @@ otp_6282/1, otp_6354/1, otp_6495/1, otp_6517/1, otp_6502/1, manpage/1, otp_6708/1, otp_7084/1, otp_7421/1, io_lib_collect_line_3_wb/1, cr_whitespace_in_string/1, - io_fread_newlines/1, otp_8989/1]). + io_fread_newlines/1, otp_8989/1, io_lib_fread_literal/1]). %-define(debug, true). @@ -62,7 +62,7 @@ all() -> otp_6282, otp_6354, otp_6495, otp_6517, otp_6502, manpage, otp_6708, otp_7084, otp_7421, io_lib_collect_line_3_wb, cr_whitespace_in_string, - io_fread_newlines, otp_8989]. + io_fread_newlines, otp_8989, io_lib_fread_literal]. groups() -> []. @@ -1995,3 +1995,29 @@ otp_8989(Suite) when is_list(Suite) -> ?line "Hel " = fmt("~-4.*s", [3,Hello]), ?line "Hel " = fmt("~*.*s", [-4,3,Hello]), ok. + +io_lib_fread_literal(doc) -> + "OTP-9439 io_lib:fread bug for literate at end"; +io_lib_fread_literal(Suite) when is_list(Suite) -> + ?line {more,"~d",0,""} = io_lib:fread("~d", ""), + ?line {error,{fread,integer}} = io_lib:fread("~d", " "), + ?line {more,"~d",1,""} = io_lib:fread(" ~d", " "), + ?line {ok,[17],"X"} = io_lib:fread(" ~d", " 17X"), + %% + ?line {more,"d",0,""} = io_lib:fread("d", ""), + ?line {error,{fread,input}} = io_lib:fread("d", " "), + ?line {more,"d",1,""} = io_lib:fread(" d", " "), + ?line {ok,[],"X"} = io_lib:fread(" d", " dX"), + %% + ?line {done,eof,_} = io_lib:fread([], eof, "~d"), + ?line {done,eof,_} = io_lib:fread([], eof, " ~d"), + ?line {more,C1} = io_lib:fread([], " \n", " ~d"), + ?line {done,{error,{fread,input}},_} = io_lib:fread(C1, eof, " ~d"), + ?line {done,{ok,[18]},""} = io_lib:fread(C1, "18\n", " ~d"), + %% + ?line {done,eof,_} = io_lib:fread([], eof, "d"), + ?line {done,eof,_} = io_lib:fread([], eof, " d"), + ?line {more,C2} = io_lib:fread([], " \n", " d"), + ?line {done,{error,{fread,input}},_} = io_lib:fread(C2, eof, " d"), + ?line {done,{ok,[]},[]} = io_lib:fread(C2, "d\n", " d"), + ok. diff --git a/lib/stdlib/test/supervisor_SUITE.erl b/lib/stdlib/test/supervisor_SUITE.erl index c79a5002fb..ff5e4c629a 100644 --- a/lib/stdlib/test/supervisor_SUITE.erl +++ b/lib/stdlib/test/supervisor_SUITE.erl @@ -114,10 +114,9 @@ end_per_group(_GroupName, Config) -> Config. init_per_testcase(count_children_memory, Config) -> - MemoryState = erlang:system_info(allocator), - case count_children_allocator_test(MemoryState) of - true -> Config; - false -> + try erlang:memory() of + _ -> Config + catch error:notsup -> {skip, "+Meamin used during test; erlang:memory/1 not available"} end; init_per_testcase(_Case, Config) -> @@ -1032,17 +1031,6 @@ count_children_memory(Config) when is_list(Config) -> [terminate(SupPid, Pid, child, kill) || {undefined, Pid, worker, _Modules} <- Children3], [1,0,0,0] = get_child_counts(sup_test). -count_children_allocator_test(MemoryState) -> - Allocators = [temp_alloc, eheap_alloc, binary_alloc, ets_alloc, - driver_alloc, sl_alloc, ll_alloc, fix_alloc, std_alloc, - sys_alloc], - MemoryStateList = element(4, MemoryState), - AllocTypes = [lists:keyfind(Alloc, 1, MemoryStateList) - || Alloc <- Allocators], - AllocStates = [lists:keyfind(e, 1, AllocValue) - || {_Type, AllocValue} <- AllocTypes], - lists:all(fun(State) -> State == {e, true} end, AllocStates). - %%------------------------------------------------------------------------- do_not_save_start_parameters_for_temporary_children(doc) -> ["Temporary children shall not be restarted so they should not " diff --git a/lib/stdlib/test/zip_SUITE.erl b/lib/stdlib/test/zip_SUITE.erl index d5f2cd52d4..7233c061ef 100644 --- a/lib/stdlib/test/zip_SUITE.erl +++ b/lib/stdlib/test/zip_SUITE.erl @@ -375,7 +375,8 @@ zip_options(Config) when is_list(Config) -> ok = file:set_cwd(?config(data_dir, Config)), %% Create a zip archive - {ok, Zip} = zip:zip("filename_not_used.zip", Names, [memory, {cwd, PrivDir}]), + {ok, {_,Zip}} = + zip:zip("filename_not_used.zip", Names, [memory, {cwd, PrivDir}]), %% Open archive {ok, ZipSrv} = zip:zip_open(Zip, [memory]), diff --git a/lib/xmerl/include/xmerl_xsd.hrl b/lib/xmerl/include/xmerl_xsd.hrl index b527accc8c..6dad7d8ff0 100644 --- a/lib/xmerl/include/xmerl_xsd.hrl +++ b/lib/xmerl/include/xmerl_xsd.hrl @@ -36,6 +36,7 @@ schema_name, vsn, schema_preprocessed=false, + external_xsd_base=false, xsd_base, xml_options=[], scope=[], diff --git a/lib/xmerl/src/xmerl_scan.erl b/lib/xmerl/src/xmerl_scan.erl index 059c8f21b6..e598c5f56d 100644 --- a/lib/xmerl/src/xmerl_scan.erl +++ b/lib/xmerl/src/xmerl_scan.erl @@ -2276,7 +2276,7 @@ scan_att_chars([H|T], S0, H, Acc, TmpAcc,AttType,IsNorm) -> % End quote true -> normalize(Acc,S,IsNorm) end, - {lists:reverse(Acc2), T, S2,IsNorm2}; + {lists:flatten(lists:reverse(Acc2)), T, S2,IsNorm2}; scan_att_chars("&" ++ T, S0, Delim, Acc, TmpAcc,AT,IsNorm) -> % Reference ?bump_col(1), {ExpRef, T1, S1} = scan_reference(T, S), diff --git a/lib/xmerl/src/xmerl_xsd.erl b/lib/xmerl/src/xmerl_xsd.erl index e56f1470c0..f003cc74ba 100644 --- a/lib/xmerl/src/xmerl_xsd.erl +++ b/lib/xmerl/src/xmerl_xsd.erl @@ -287,10 +287,19 @@ process_schema(Schema) -> %% error reason. The error reason may be a list of several errors %% or a single error encountered during the processing. process_schema(Schema,Options) when is_list(Options) -> - S = initiate_state(Options,Schema), - process_schema2(xmerl_scan:file(filename:join(S#xsd_state.xsd_base, Schema)),S,Schema); -process_schema(Schema,State) when is_record(State,xsd_state) -> - process_schema2(xmerl_scan:file(filename:join(State#xsd_state.xsd_base, Schema)),State,Schema). + State = initiate_state(Options,Schema), + process_schema(Schema, State); +process_schema(Schema, State=#xsd_state{fetch_fun=Fetch})-> + case Fetch(Schema, State) of + {ok,{file,File},_} -> + process_schema2(xmerl_scan:file(File), State, Schema); + {ok,{string,Str},_} -> + process_schema2(xmerl_scan:string(Str), State, Schema); + {ok,[],_} -> + {error,enoent}; + Err -> + Err + end. process_schema2(Err={error,_},_,_) -> Err; @@ -319,12 +328,9 @@ process_schemas(Schemas) -> %% error reason. The error reason may be a list of several errors %% or a single error encountered during the processing. process_schemas(Schemas=[{_,Schema}|_],Options) when is_list(Options) -> - process_schemas(Schemas,initiate_state(Options,Schema)); + State = initiate_state(Options,Schema), + process_schemas(Schemas, State); process_schemas([{_NS,Schema}|Rest],State=#xsd_state{fetch_fun=Fetch}) -> -%% case process_external_schema_once(Schema,if_list_to_atom(NS),State) of -%% S when is_record(S,xsd_state) -> -%% case process_schema(filename:join([State#xsd_state.xsd_base,Schema]),State) of -%% {ok,S} -> Res= case Fetch(Schema,State) of {ok,{file,File},_} -> @@ -345,20 +351,20 @@ process_schemas([{_NS,Schema}|Rest],State=#xsd_state{fetch_fun=Fetch}) -> process_schemas([],S) when is_record(S,xsd_state) -> {ok,S}. - initiate_state(Opts,Schema) -> XSDBase = filename:dirname(Schema), {{state,S},RestOpts}=new_state(Opts), S2 = create_tables(S), - initiate_state2(S2#xsd_state{schema_name = Schema, - xsd_base = XSDBase, - fetch_fun = fun fetch/2},RestOpts). + S3 = initiate_state2(S2#xsd_state{schema_name = Schema, xsd_base=XSDBase, + fetch_fun = fun fetch/2}, + RestOpts). + initiate_state2(S,[]) -> S; initiate_state2(S,[{tab2file,Bool}|T]) -> initiate_state2(S#xsd_state{tab2file=Bool},T); -initiate_state2(S,[{xsdbase,XSDBase}|T]) -> - initiate_state2(S#xsd_state{xsd_base=XSDBase},T); +initiate_state2(S,[{xsdbase, XSDBase}|T]) -> + initiate_state2(S#xsd_state{xsd_base=XSDBase, external_xsd_base=true},T); initiate_state2(S,[{fetch_fun,FetchFun}|T]) -> initiate_state2(S#xsd_state{fetch_fun=FetchFun},T); initiate_state2(S,[{fetch_path,FetchPath}|T]) -> @@ -5232,7 +5238,12 @@ fetch(URI,S) -> [] -> %% empty systemliteral []; _ -> - filename:join(S#xsd_state.xsd_base, URI) + case S#xsd_state.external_xsd_base of + true -> + filename:join(S#xsd_state.xsd_base, URI); + false -> + filename:join(S#xsd_state.xsd_base, filename:basename(URI)) + end end, Path = path_locate(S#xsd_state.fetch_path, Filename, Fullname), ?dbg("fetch(~p) -> {file, ~p}.~n", [URI, Path]), diff --git a/lib/xmerl/test/xmerl_SUITE.erl b/lib/xmerl/test/xmerl_SUITE.erl index 392b2522e8..0c809dbcb6 100644 --- a/lib/xmerl/test/xmerl_SUITE.erl +++ b/lib/xmerl/test/xmerl_SUITE.erl @@ -57,7 +57,8 @@ groups() -> {eventp_tests, [], [sax_parse_and_export]}, {ticket_tests, [], [ticket_5998, ticket_7211, ticket_7214, ticket_7430, - ticket_6873, ticket_7496, ticket_8156, ticket_8697]}, + ticket_6873, ticket_7496, ticket_8156, ticket_8697, + ticket_9411]}, {app_test, [], [{xmerl_app_test, all}]}, {appup_test, [], [{xmerl_appup_test, all}]}]. @@ -575,7 +576,17 @@ ticket_8697(Config) -> ?line [16#545C] = HexEntityText, ok. +ticket_9411(suite) -> []; +ticket_9411(doc) -> + ["Test that xmerl_scan handles attribute that contains for example ""]; +ticket_9411(Config) -> + DataDir = ?config(data_dir,Config), + ?line {ok, Schema} = xmerl_xsd:process_schema(filename:join([DataDir,"misc/ticket_9411.xsd"])), + ?line {ok, Bin} = file:read_file(filename:join([DataDir,"misc/ticket_9411.xml"])), + ?line Xml = erlang:binary_to_list(Bin), + ?line {E, _} = xmerl_scan:string(Xml), + ?line {E, _} = xmerl_xsd:validate(E, Schema). diff --git a/lib/xmerl/test/xmerl_SUITE_data/misc.tar.gz b/lib/xmerl/test/xmerl_SUITE_data/misc.tar.gz Binary files differindex c48a6f897b..fef7431845 100644 --- a/lib/xmerl/test/xmerl_SUITE_data/misc.tar.gz +++ b/lib/xmerl/test/xmerl_SUITE_data/misc.tar.gz diff --git a/lib/xmerl/test/xmerl_xsd_SUITE.erl b/lib/xmerl/test/xmerl_xsd_SUITE.erl index a0d3b1e667..421fa48054 100644 --- a/lib/xmerl/test/xmerl_xsd_SUITE.erl +++ b/lib/xmerl/test/xmerl_xsd_SUITE.erl @@ -62,7 +62,7 @@ groups() -> sis2, state2file_file2state, union]}, {ticket_tests, [], [ticket_6910, ticket_7165, ticket_7190, ticket_7288, - ticket_7736, ticket_8599]}, + ticket_7736, ticket_8599, ticket_9410]}, {facets, [], [length, minLength, maxLength, pattern, enumeration, whiteSpace, maxInclusive, maxExclusive, minExclusive, @@ -1146,3 +1146,8 @@ ticket_8599(Config) -> ?line {{xmlElement,persons,persons,_,_,_,_,_,_,_,_,_},_GlobalState} = xmerl_xsd:validate(E, S). + +ticket_9410(suite) -> []; +ticket_9410(Config) -> + file:set_cwd(filename:join([?config(data_dir,Config),".."])), + ?line {ok, _S} = xmerl_xsd:process_schema("xmerl_xsd_SUITE_data/small.xsd"). |