diff options
46 files changed, 580 insertions, 128 deletions
diff --git a/erts/doc/src/notes.xml b/erts/doc/src/notes.xml index ff3c798179..25b72ce774 100644 --- a/erts/doc/src/notes.xml +++ b/erts/doc/src/notes.xml @@ -31,6 +31,29 @@ </header> <p>This document describes the changes made to the ERTS application.</p> +<section><title>Erts 9.1.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>Added zlib:set_controlling_process/2 to move a + zstream() between processes.</p> + <p> + Own Id: OTP-14672 Aux Id: ERL-494 </p> + </item> + <item> + <p> + Fix so that schedulers are bound correctly when the first + available cpu is not the first detected cpu. e.g. when + using "taskset -c X..Y" when X is not equal to 0.</p> + <p> + Own Id: OTP-14694</p> + </item> + </list> + </section> + +</section> + <section><title>Erts 9.1.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/erts/doc/src/zlib.xml b/erts/doc/src/zlib.xml index f5cc1b1e64..6f4c42da27 100644 --- a/erts/doc/src/zlib.xml +++ b/erts/doc/src/zlib.xml @@ -71,6 +71,11 @@ list_to_binary([Compressed|Last])</pre> called prior to a call to <seealso marker="#inflate/2"><c>inflate/2</c></seealso>. </item> + <tag><c>not_on_controlling_process</c></tag> + <item>The stream was used by a process that doesn't control it. Use + <seealso marker="#set_controlling_process/2"> + <c>set_controlling_process/2</c></seealso> if you need to transfer + a stream to a different process.</item> <tag><c>data_error</c></tag> <item>The data contains errors. </item> @@ -740,6 +745,15 @@ loop(Z, Handler, {finished, Output}) -> </func> <func> + <name name="set_controlling_process" arity="2"/> + <fsummary>Transfers ownership of a zlib stream.</fsummary> + <desc> + <p>Changes the controlling process of <c><anno>Z</anno></c> to + <c><anno>Pid</anno></c>, which must be a local process.</p> + </desc> + </func> + + <func> <name name="uncompress" arity="1"/> <fsummary>Uncompress data with standard zlib functionality.</fsummary> <desc> diff --git a/erts/emulator/beam/erl_cpu_topology.c b/erts/emulator/beam/erl_cpu_topology.c index 49f9beb19f..6f8d2f8c35 100644 --- a/erts/emulator/beam/erl_cpu_topology.c +++ b/erts/emulator/beam/erl_cpu_topology.c @@ -602,7 +602,7 @@ write_schedulers_bind_change(erts_cpu_topology_t *cpudata, int size) cpu_bind_order_sort(cpudata, size, cpu_bind_order, 1); - for (cpu_ix = 0; cpu_ix < size && cpu_ix < erts_no_schedulers; cpu_ix++) + for (cpu_ix = 0; cpu_ix < size && s_ix <= erts_no_schedulers; cpu_ix++) if (erts_is_cpu_available(cpuinfo, cpudata[cpu_ix].logical)) scheduler2cpu_map[s_ix++].bind_id = cpudata[cpu_ix].logical; } diff --git a/erts/emulator/beam/erl_process_dump.c b/erts/emulator/beam/erl_process_dump.c index 12ef4aab8a..0b7f361622 100644 --- a/erts/emulator/beam/erl_process_dump.c +++ b/erts/emulator/beam/erl_process_dump.c @@ -728,8 +728,15 @@ static void mark_literal(Eterm* ptr) ap = bsearch(ptr, lit_areas, num_lit_areas, sizeof(ErtsLiteralArea*), search_areas); - ASSERT(ap); - ap[0]->off_heap = (struct erl_off_heap_header *) 1; + + /* + * If the literal was created by native code, this search will not + * find it and ap will be NULL. + */ + + if (ap) { + ap[0]->off_heap = (struct erl_off_heap_header *) 1; + } } diff --git a/erts/emulator/internal_doc/GarbageCollection.md b/erts/emulator/internal_doc/GarbageCollection.md new file mode 100644 index 0000000000..1d9e3f4160 --- /dev/null +++ b/erts/emulator/internal_doc/GarbageCollection.md @@ -0,0 +1,187 @@ +# Erlang Garbage Collector + +Erlang manages dynamic memory with a [tracing garbage collector](https://en.wikipedia.org/wiki/Tracing_garbage_collection). More precisely a per process generational semi-space copying collector using [Cheney's](#cheney) copy collection algorithm together with a global large object space. + +## Overview + +Each Erlang process has its own stack and heap which are allocated in the same memory block and grow towards each other. When the stack and the heap [meet](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/beam_emu.c#L387), the garbage collector is triggered and memory is reclaimed. If not enough memory was reclaimed, the heap will grow. + +### Creating Data + +Terms are created on the heap by evaluating expressions. There are two major types of terms: [immediate terms](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_term.h#L88-L97) which require no heap space (small integers, atoms, pids, port ids etc) and cons or [boxed terms](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_term.h#L106-L120) (tuple, big num, binaries etc) that do require heap space. Immediate terms do not need any heap space because they are embedded into the containing structure. + +Let's look at an example that returns a tuple with the newly created data. + +```erlang +data(Foo) -> + Cons = [42|Foo], + Literal = {text, "hello world!"}, + {tag, Cons, Literal}. +``` + +In this example we first create a new cons cell with an integer and a tuple with some text. Then a tuple of size three wrapping the other values with an atom tag is created and returned. + +On the heap tuples require a word size for each of its elements as well as for the header. Cons cells always require two words. Adding these things together, we get seven words for the tuples and 26 words for the cons cells. The string `"hello world!"` is a list of cons cells and thus requires 24 words. The atom `tag` and the integer `42` do not require any additional heap memory since it is an *immediate*. Adding all the terms together, the heap space required in this example should be 33 words. + +Compiling this code to beam assembly (`erlc -S`) shows exactly what is happening. + +```erlang + ... + {test_heap,6,1}. + {put_list,{integer,42},{x,0},{x,1}}. + {put_tuple,3,{x,0}}. + {put,{atom,tag}}. + {put,{x,1}}. + {put,{literal,{text,"hello world!"}}}. + return. +``` + +Looking at the assembler code we can see three things; The heap requirement in this function turns out to be only six words, as seen by the `{test_heap,6,1}` instruction. All the allocations are combined to a single instruction. The bulk of the data `{text, "hello world!"}` is a *literal*. Literals, sometimes referred to as constants, are not allocated in the function since they are a part of the module and allocated at load time. + +If there is not enough space available on the heap to satisfy the `test_heap` instructions request for memory, then a garbage collection is initiated. It may happen immediately in the `test_heap` instruction, or it can be delayed until a later time depending on what state the process is in. If the garbage collection is delayed, any memory needed will be allocated in heap fragments. Heap fragments are extra memory blocks that are a part of the young heap, but are not allocated in the contigious area where terms normally reside. See [The young heap](#the-young-heap) for more details. + +### The collector + +Erlang has a copying semi-space garbage collector. This means that when doing a garbage collection, the terms are copied from one distinct area, called the *from space*, to a new clean area, called the *to space*. The collector starts by [scanning the root-set](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1980) (stack, registers, etc). + + + +It follows all the pointers from the root-set to the heap and copies each term word by word to the *to space*. + +After the header word has been copied a [*move marker*](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.h#L45-L46) is destructively placed in it pointing to the term in the *to space*. Any other term that points to the already moved term will [see this move marker](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1125) and copy the referring pointer instead. For example, if the have the following Erlang code: + +```erlang +foo(Arg) -> + T = {test, Arg}, + {wrapper, T, T, T}. +``` + +Only one copy of T exists on the heap and during the garbage collection only the first time T is encountered will it be copied. + + + +After [all terms](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1089) referenced by the root-set have been copied, the collector scans the *to space* and copies all terms that these terms reference. When scanning, the collector steps through each term on the *to space* and any term still referencing the *from space* is copied over to the *to space*. Some terms contain non-term data (the payload of a on heap binary for instance). When encountered by the collector, these values are simply skipped. + + + +Every term object we can reach is copied to the *to space* and stored on top off the *scan stop* line, and then the scan stop is moved to the end of the last object. + + + +When *scan stop* marker [catches up](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1103) to the *scan start* marker, the garbage collection is done. At this point we can [deallocate](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1206) the entire *from space* and therefore reclaim the entire young heap. + +## Generational Garbage Collection + +In addition to the collection algorithm described above, the Erlang garbage collector also provides generational garbage collection. An additional heap, called the old heap, is used where the long lived data is stored. The original heap is called the young heap, or sometimes the allocation heap. + +With this in mind we can look at the Erlang's garbage collection again. During the copy stage anything that should be copied to the young *to space* is instead copied to the old *to space* *if* it is [below the *high-watermark*](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1127). + + + +The [*high-watermark*](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_process.h#L1021) is placed where the previous garbage collection (described in [Overview](#overview)) ended and we have introduced a new area called the old heap. When doing the normal garbage collection pass, any term that is located below the high-watermark is copied to the old *to space* instead of the young. + + + +In the next garbage collection, any pointers to the old heap will be ignored and not scanned. This way the garbage collector does not have to scan the long-lived terms. + +Generational garbage collection aims to increase performance at the expense of memory. This is achieved because only the young, smaller, heap is considered in most garbage collections. + +The generational [hypothesis](#ungar) predicts that most terms tend to die young, and for an immutable language such as Erlang, young terms die even faster than in other languages. So for most usage patterns the data in the new heap will die very soon after it is allocated. This is good because it limits the amount of data copied to the old heap and also because the garbage collection algorithm used is proportional to the amount of live data on the heap. + +One critical issue to note here is that any term on the young heap can reference terms on the old heap but *no* term on the old heap may refer to a term on the young heap. This is due to the nature of the copy algorithm. Anything referenced by an old heap term is not included in the reference tree, root-set and its followers, and hence is not copied. If it was, the data would be lost, fire and brimstone would rise to cover the earth. Fortunately, this comes naturally for Erlang because the terms are immutable and thus there can be no pointers modified on the old heap to point to the young heap. + +To reclaim data from the old heap, both young and old heaps are included during the collection and copied to a common *to space*. Both the *from space* of the young and old heap are then deallocated and the procedure will start over from the beginning. This type of garbage collection is called a full sweep and is triggered when the size of the area under the high-watermark is larger than the size of the free area of the old heap. It can also be triggered by doing a manual call to [erlang:garbage_collect()](http://erlang.org/doc/man/erlang.html#garbage_collect-0), or by running into the young garbage collection limit set by [spawn_opt(fun(),[{fullsweep_after, N}])](http://erlang.org/doc/man/erlang.html#spawn_opt-4) where N is the number of young garbage collections to do before forcing a garbage collection of both young and old heap. + +## The young heap + +The young heap, or the allocation heap, consists of the stack and heap as described in the Overview. However, it also includes any heap fragments that are attached to the heap. All of the heap fragments are considered to be above the high-watermark and part of the young generation. Heap fragments contain terms that either did not fit on the heap, or were created by another process and then attached to the heap. For instance if the bif binary_to_term created a term which does not fit on the current heap without doing a garbage collection, it will create a heap-fragment for the term and then schedule a garbage collection for later. Also if a message is sent to the process, the payload may be placed in a heap-fragment and that fragment is added to young heap when the message is matched in a receive clause. + +This procedure differs from how it worked prior to Erlang/OTP 19.0. Before 19.0, only a contiguous memory block where the young heap and stack resided was considered to be part of the young heap. Heap fragments and messages were immediately copied into the young heap before they could be inspected by the Erlang program. The behaviour introduced in 19.0 is superior in many ways - most significantly it reduces the number of necessary copy operations and the root set for garbage collection. + +## Sizing the heap + +As mentioned in the Overview the size of the heap [grows](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L247) to accommodate more data. Heaps grow in two stages, first a [variation of the Fibonacci sequence](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L199-L208) is used starting at 233 words. Then at about 1 mega words the heap only [grows in 20% increments](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L215-L227). + +There are two occasions when the young heap grows: + +* if the total size of the heap + message and heap fragments exceeds the current heap size. +* if after a fullsweep, the total amount of live objects is greater than 75%. + +There are two occasions when the young heap is shrunk: + +* if after a young collection, the total amount of live objects is less than 25% of the heap and the young heap is "big" +* if after a fullsweep, the total amount of live objects is less than 25% of the heap. + +The old heap is always one step ahead in the heap growth stages than the young heap. + +## Literals + +When garbage collecting a heap (young or old) all literals are left in place and not copied. To figure out if a term should be copied or not when doing a garbage collection the following pseudo code is used: + +```c +if (erts_is_literal(ptr) || (on_old_heap(ptr) && !fullsweep)) { + /* literal or non fullsweep - do not copy */ +} else { + copy(ptr); +} +``` + +The [`erts_is_literal`](https://github.com/erlang/otp/blob/OTP-19.0/erts/emulator/beam/global.h#L1452-L1465) check works differently on different architectures and operating systems. + +On 64 bit systems that allow mapping of unreserved virtual memory areas (most operating systems except Windows), an area of size 1 GB (by default) is mapped and then all literals are placed within that area. Then all that has to be done to determine if something is a literal or not is [two quick pointer checks](https://github.com/erlang/otp/blob/OTP-19.0/erts/emulator/beam/erl_alloc.h#L322-L324). This system relies on the fact that a memory page that has not been touched yet does not take any actual space. So even if 1 GB of virtual memory is mapped, only the memory which is actually needed for literals is allocated in ram. The size of the literal area is configurable through the +MIscs erts_alloc option. + +On 32 bit systems, there is not enough virtual memory space to allocate 1 GB for just literals, so instead small 256 KB sized literal regions are created on demand and a card mark bit-array of the entire 32 bit memory space is then used to determine if a term is a literal or not. Since the total memory space is only 32 bits, the card mark bit-array is only 256 words large. On a 64 bit system the same bit-array would have to be 1 tera words large, so this technique is only viable on 32 bit systems. Doing [lookups in the array](https://github.com/erlang/otp/blob/OTP-19.0/erts/emulator/beam/erl_alloc.h#L316-L319) is a little more expensive then just doing the pointer checks that can be done in 64 bit systems, but not extremely so. + +On 64 bit windows, on which erts_alloc cannot do unreserved virtual memory mappings, a [special tag](https://github.com/erlang/otp/blob/OTP-19.0/erts/emulator/beam/erl_term.h#L59) within the Erlang term object is used to determine if something [is a literal or not](https://github.com/erlang/otp/blob/OTP-19.0/erts/emulator/beam/erl_term.h#L248-L252). This is very cheap, however, the tag is only available on 64 bit machines, and it is possible to do a great deal of other nice optimizations with this tag in the future (like for instance a more compact list implementation) so it is not used on operating systems where it is not needed. + +This behaviour is different from how it worked prior to Erlang/OTP 19.0. Before 19.0 the literal check was done by checking if the pointer pointed to the young or old heap block. If it did not, then it was considered a literal. This lead to considerable overhead and strange memory usage scenarios, so it was removed in 19.0. + +## Binary heap + +The binary heap works as a large object space for binary terms that are greater than 64 bytes (from now on called off-heap binaries). The binary heap is [reference counted](https://en.wikipedia.org/wiki/Reference_counting) and a pointer to the off-heap binary is stored on the process heap. To keep track of when to decrement the reference counter of the off-heap binary, a linked list (the MSO - mark and sweep object list) containing funs and externals as well as off-heap binaries is woven through the heap. After a garbage collection is done, the [MSO list is swept](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2299) and any off-heap binary that does not have a [move marker](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2325) written into the header words has its reference [decremented and is potentially freed](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2344-L2367). + +All items in the MSO list are ordered by the time they were added to the process heap, so when doing a minor garbage collection, the MSO sweeper only has to sweep until it [encounters an off-heap binary that is on the old heap](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2369). + +### Virtual Binary heap + +Each process has a virtual binary heap associated with it that has the size of all the current off-heap binaries that the process has references to. The virtual binary heap also has a limit and grows and shrinks depending on how off-heap binaries are used by the process. The same growth and shrink mechanisms are used for the binary heap and for the term heap, so first a Fibonacci like series and then 20% growth. + +The virtual binary heap exists in order to [trigger](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/beam_emu.c#L364) garbage collections earlier when potentially there is a very large amount of off-heap binary data that could be reclaimed. This approach does not catch all problems with binary memory not being released soon enough, but it does catch a lot of them. + +## Messages + +Messages can become a part of the process heap at different times. This depends on how the process is configured. +We can configure the behaviour of each process using `process_flag(message_queue_data, off_heap | on_heap)` or we can set a default for all processes at start using the option `+hmqd`. + +What do these different configurations do and when should we use them? +Let's start by going through what happens when one Erlang process sends a message to another. +The sending process needs to do a couple of things: + +1. calculate [how large](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L1031) the message to be sent is +2. [allocate enough space](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L1033) to fit the entire message +3. [copy](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L1040) the message payload +4. allocate a message container with some meta data +5. [insert](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L502) the message container in the receiver process' [message queue](https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_process.h#L1042) + +The process flag `message_queue_data`, of the receiver process, controls the message allocating strategy of the sender process in step 2 and also how the message data is treated by the garbage collector. + +The procedure above is different from how it worked prior to 19.0. Before 19.0 there was no configuration option, the behaviour was always very similar to how the `on_heap` option is in 19.0. + +### Message allocating strategies + +If set to `on_heap`, the sending process will first attempt to allocate the space for the message directly on the young heap block of the receiving process. +This is not always possible as it requires taking the *main lock* of the receiving process. The main lock is also held when the process is executing. The possibility for a lock conflict is thus likely in an intensely collaborating system. +If the sending process cannot acquire the main lock, a heap fragment is instead created for the message and the message payload is copied onto that. +With the `off_heap` option the sender process always creates heap fragments for messages sent to that process. + +There are a bunch of different tradeoffs that come into play when trying to figure out which of the strategies you want to use. + +Using `off_heap` may seem like a nice way to get a more scalable system as you get very little contention on the main locks, however, allocating a heap fragment is more expensive than allocating on the heap of the receiving process. So if it is very unlikely that contention will occur, it is more efficient to try to allocate the message directly on the receiving process' heap. + +Using `on_heap` will force all messages to be part of on the young heap which will increase the amount of data that the garbage collector has to move. So if a garbage collection is triggered while processing a large amount of messages, they will be copied to the young heap. This in turn will lead to that the messages will quickly be promoted to the old heap and thus increase its size. This may be good or bad depending on exactly what the process does. A large old heap means that the young heap will also be larger, which in turn means that less garbage collections will be triggered while processing the message queue. This will temporarly increase the throughput of the process at the cost of more memory usage. However, if after all the messages have been consumed the process enters a state where a lot less messages are being received. Then it may be a long time before the next fullsweep garbage collection happens and the messages that are on the old heap will be there until that happens. So while `on_heap` is potentially faster than the other modes, it uses more memory for a longer time. This mode is the legacy mode which is almost how the message queue was handled before Erlang/OTP 19.0. + +Which one of these strategies is best depends a lot on what the process is doing and how it interacts with other processes. So, as always, profile the application and see how it behaves with the different options. + + <a name="cheney">[1]</a>: C. J. Cheney. A nonrecursive list compacting algorithm. Commun. ACM, 13(11):677–678, Nov. 1970. + + <a name="ungar">[2]</a>: D. Ungar. Generation scavenging: A non-disruptive high performance storage reclamation algorithm. SIGSOFT Softw. Eng. Notes, 9(3):157–167, Apr. 1984. diff --git a/erts/emulator/internal_doc/figures/.gitignore b/erts/emulator/internal_doc/figures/.gitignore new file mode 100644 index 0000000000..c2813ac866 --- /dev/null +++ b/erts/emulator/internal_doc/figures/.gitignore @@ -0,0 +1 @@ +*.eps
\ No newline at end of file diff --git a/erts/emulator/internal_doc/figures/Makefile b/erts/emulator/internal_doc/figures/Makefile new file mode 100644 index 0000000000..111cb393fb --- /dev/null +++ b/erts/emulator/internal_doc/figures/Makefile @@ -0,0 +1,20 @@ +# In order to update the figures you have to have both dia +# and imagemagick installed. + +DIAGRAMS=$(wildcard *.dia) +EPS_DIAGRAMS=$(patsubst %.dia,%.eps,$(DIAGRAMS)) +PNG_DIAGRAMS=$(patsubst %.dia,%.png,$(DIAGRAMS)) + +diagrams: $(EPS_DIAGRAMS) + +png: $(PNG_DIAGRAMS) + +update_png: png + git add $(PNG_DIAGRAMS) + git commit -m "Update internal docs figures" + +%.eps: %.dia + dia --export=$@ $< + +%.png: %.eps + convert $< -resize 65% $@ diff --git a/erts/emulator/internal_doc/figures/gc-heap-scan1.dia b/erts/emulator/internal_doc/figures/gc-heap-scan1.dia Binary files differnew file mode 100644 index 0000000000..b8a32dc092 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-heap-scan1.dia diff --git a/erts/emulator/internal_doc/figures/gc-heap-scan1.png b/erts/emulator/internal_doc/figures/gc-heap-scan1.png Binary files differnew file mode 100644 index 0000000000..9724fc8698 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-heap-scan1.png diff --git a/erts/emulator/internal_doc/figures/gc-heap-stop.dia b/erts/emulator/internal_doc/figures/gc-heap-stop.dia Binary files differnew file mode 100644 index 0000000000..af219958c6 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-heap-stop.dia diff --git a/erts/emulator/internal_doc/figures/gc-heap-stop.png b/erts/emulator/internal_doc/figures/gc-heap-stop.png Binary files differnew file mode 100644 index 0000000000..ec79790d36 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-heap-stop.png diff --git a/erts/emulator/internal_doc/figures/gc-rootset-scan.dia b/erts/emulator/internal_doc/figures/gc-rootset-scan.dia Binary files differnew file mode 100644 index 0000000000..d6147740e5 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-rootset-scan.dia diff --git a/erts/emulator/internal_doc/figures/gc-rootset-scan.png b/erts/emulator/internal_doc/figures/gc-rootset-scan.png Binary files differnew file mode 100644 index 0000000000..06509f83c3 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-rootset-scan.png diff --git a/erts/emulator/internal_doc/figures/gc-start.dia b/erts/emulator/internal_doc/figures/gc-start.dia Binary files differnew file mode 100644 index 0000000000..41832d29b0 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-start.dia diff --git a/erts/emulator/internal_doc/figures/gc-start.png b/erts/emulator/internal_doc/figures/gc-start.png Binary files differnew file mode 100644 index 0000000000..6c29e9adb0 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-start.png diff --git a/erts/emulator/internal_doc/figures/gc-watermark-2.dia b/erts/emulator/internal_doc/figures/gc-watermark-2.dia Binary files differnew file mode 100644 index 0000000000..2ea3be3fb2 --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-watermark-2.dia diff --git a/erts/emulator/internal_doc/figures/gc-watermark-2.png b/erts/emulator/internal_doc/figures/gc-watermark-2.png Binary files differnew file mode 100644 index 0000000000..bca110282d --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-watermark-2.png diff --git a/erts/emulator/internal_doc/figures/gc-watermark.dia b/erts/emulator/internal_doc/figures/gc-watermark.dia Binary files differnew file mode 100644 index 0000000000..a5de7565dc --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-watermark.dia diff --git a/erts/emulator/internal_doc/figures/gc-watermark.png b/erts/emulator/internal_doc/figures/gc-watermark.png Binary files differnew file mode 100644 index 0000000000..b835c7694e --- /dev/null +++ b/erts/emulator/internal_doc/figures/gc-watermark.png diff --git a/erts/emulator/nifs/common/zlib_nif.c b/erts/emulator/nifs/common/zlib_nif.c index a9c5b05e47..fa29b4fb71 100644 --- a/erts/emulator/nifs/common/zlib_nif.c +++ b/erts/emulator/nifs/common/zlib_nif.c @@ -106,6 +106,7 @@ typedef struct { int inflateChunk_buffer_size; ErlNifPid controlling_process; + ErlNifMutex *controller_lock; ErlNifIOQueue *input_queue; @@ -117,6 +118,8 @@ typedef struct { static ERL_NIF_TERM zlib_open(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); static ERL_NIF_TERM zlib_close(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); +static ERL_NIF_TERM zlib_set_controller(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); + static ERL_NIF_TERM zlib_deflateInit(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); static ERL_NIF_TERM zlib_deflateSetDictionary(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); static ERL_NIF_TERM zlib_deflateReset(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); @@ -143,6 +146,11 @@ static ERL_NIF_TERM zlib_setBufSize(ErlNifEnv *env, int argc, const ERL_NIF_TERM static ERL_NIF_TERM zlib_enqueue_input(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]); static ErlNifFunc nif_funcs[] = { + {"close_nif", 1, zlib_close}, + {"open_nif", 0, zlib_open}, + + {"set_controller_nif", 2, zlib_set_controller}, + /* deflate */ {"deflateInit_nif", 6, zlib_deflateInit}, {"deflateSetDictionary_nif", 2, zlib_deflateSetDictionary}, @@ -162,10 +170,6 @@ static ErlNifFunc nif_funcs[] = { /* running checksum */ {"crc32_nif", 1, zlib_crc32}, - /* open & close */ - {"close_nif", 1, zlib_close}, - {"open_nif", 0, zlib_open}, - /* The stash keeps a single term alive across calls, and is used in * exception_on_need_dict/1 to retain the old error behavior, and for * saving data flushed through deflateParams/3. */ @@ -281,9 +285,7 @@ static ERL_NIF_TERM zlib_return(ErlNifEnv *env, int code) { return reason; } -static void gc_zlib(ErlNifEnv *env, void* data) { - zlib_data_t *d = (zlib_data_t*)data; - +static void zlib_internal_close(zlib_data_t *d) { if(d->state == ST_DEFLATE) { deflateEnd(&d->s); } else if(d->state == ST_INFLATE) { @@ -291,8 +293,6 @@ static void gc_zlib(ErlNifEnv *env, void* data) { } if(d->state != ST_CLOSED) { - enif_ioq_destroy(d->input_queue); - if(d->stash_env != NULL) { enif_free_env(d->stash_env); } @@ -301,17 +301,36 @@ static void gc_zlib(ErlNifEnv *env, void* data) { } } +static void gc_zlib(ErlNifEnv *env, void* data) { + zlib_data_t *d = (zlib_data_t*)data; + + enif_mutex_destroy(d->controller_lock); + enif_ioq_destroy(d->input_queue); + + zlib_internal_close(d); + + (void)env; +} + static int get_zlib_data(ErlNifEnv *env, ERL_NIF_TERM opaque, zlib_data_t **d) { return enif_get_resource(env, opaque, rtype_zlib, (void **)d); } static int zlib_process_check(ErlNifEnv *env, zlib_data_t *d) { + int is_controlling_process; ErlNifPid current_process; enif_self(env, ¤t_process); - return enif_is_identical(enif_make_pid(env, ¤t_process), + enif_mutex_lock(d->controller_lock); + + is_controlling_process = enif_is_identical( + enif_make_pid(env, ¤t_process), enif_make_pid(env, &d->controlling_process)); + + enif_mutex_unlock(d->controller_lock); + + return is_controlling_process; } static void zlib_reset_input(zlib_data_t *d) { @@ -516,6 +535,8 @@ static ERL_NIF_TERM zlib_open(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[ d->input_queue = enif_ioq_create(ERL_NIF_IOQ_NORMAL); + d->controller_lock = enif_mutex_create("zlib_controller_lock"); + d->s.zalloc = zlib_alloc; d->s.zfree = zlib_free; d->s.opaque = d; @@ -556,7 +577,28 @@ static ERL_NIF_TERM zlib_close(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv return enif_raise_exception(env, am_not_initialized); } - gc_zlib(env, d); + zlib_internal_close(d); + + return am_ok; +} + +static ERL_NIF_TERM zlib_set_controller(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { + zlib_data_t *d; + + ErlNifPid new_owner; + + if(argc != 2 || !get_zlib_data(env, argv[0], &d) + || !enif_get_local_pid(env, argv[1], &new_owner)) { + return enif_make_badarg(env); + } else if(!zlib_process_check(env, d)) { + return enif_raise_exception(env, am_not_on_controlling_process); + } + + enif_mutex_lock(d->controller_lock); + + d->controlling_process = new_owner; + + enif_mutex_unlock(d->controller_lock); return am_ok; } diff --git a/erts/preloaded/ebin/zlib.beam b/erts/preloaded/ebin/zlib.beam Binary files differindex 5048bdb846..f388bc723a 100644 --- a/erts/preloaded/ebin/zlib.beam +++ b/erts/preloaded/ebin/zlib.beam diff --git a/erts/preloaded/src/zlib.erl b/erts/preloaded/src/zlib.erl index 3170ab6351..03c9ae38a1 100644 --- a/erts/preloaded/src/zlib.erl +++ b/erts/preloaded/src/zlib.erl @@ -20,7 +20,8 @@ -module(zlib). --export([open/0,close/1,deflateInit/1,deflateInit/2,deflateInit/6, +-export([open/0,close/1,set_controlling_process/2, + deflateInit/1,deflateInit/2,deflateInit/6, deflateSetDictionary/2,deflateReset/1,deflateParams/3, deflate/2,deflate/3,deflateEnd/1, inflateInit/1,inflateInit/2,inflateInit/3, @@ -128,6 +129,14 @@ close(Z) -> close_nif(_Z) -> erlang:nif_error(undef). +-spec set_controlling_process(Z, Pid) -> 'ok' when + Z :: zstream(), + Pid :: pid(). +set_controlling_process(Z, Pid) -> + set_controller_nif(Z, Pid). +set_controller_nif(_Z, _Pid) -> + erlang:nif_error(undef). + -spec deflateInit(Z) -> 'ok' when Z :: zstream(). deflateInit(Z) -> diff --git a/erts/vsn.mk b/erts/vsn.mk index 3d7ff2db66..a788b2e491 100644 --- a/erts/vsn.mk +++ b/erts/vsn.mk @@ -18,7 +18,7 @@ # %CopyrightEnd% # -VSN = 9.1.2 +VSN = 9.1.3 # Port number 4365 in 4.2 # Port number 4366 in 4.3 diff --git a/lib/diameter/doc/src/notes.xml b/lib/diameter/doc/src/notes.xml index 589e7d5145..eded788419 100644 --- a/lib/diameter/doc/src/notes.xml +++ b/lib/diameter/doc/src/notes.xml @@ -43,6 +43,23 @@ first.</p> <!-- ===================================================================== --> +<section><title>diameter 2.1.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + A fault introduced in diameter 2.1 could cause decode + errors to be ignored in AVPs following the header of a + Grouped AVP.</p> + <p> + Own Id: OTP-14684 Aux Id: ERIERL-85 </p> + </item> + </list> + </section> + +</section> + <section><title>diameter 2.1.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/diameter/src/base/diameter_gen.erl b/lib/diameter/src/base/diameter_gen.erl index d3b9f704fe..93ebe57685 100644 --- a/lib/diameter/src/base/diameter_gen.erl +++ b/lib/diameter/src/base/diameter_gen.erl @@ -320,8 +320,8 @@ decode(Bin, Code, Vid, DataLen, Pad, M, P, Name, Mod, Fmt, Strict, Opts0, index = Idx}, Dec = dec(Data, Name, NameT, Mod, Fmt, Opts, Avp), - Acc = decode(T, Name, Mod, Fmt, Strict, Opts, Idx+1, AM),%% recurse - acc(Acc, Dec, I, Field, Arity, Strict, Mod, Opts); + Acc = decode(T, Name, Mod, Fmt, Strict, Opts0, Idx+1, AM),%% recurse + acc(Acc, Dec, I, Field, Arity, Strict, Mod); _ -> {NameT, _Field, _Arity, {_, AM}} = incr(Name, Code, Vid, M, Mod, Strict, Opts0, AM0), @@ -574,15 +574,17 @@ dec_AVP(Dicts, Data, Name, Mod, Fmt, Opts, #diameter_avp{code = Code, %% Try to decode an AVP in the first alternate dictionary that defines %% it. -dec_AVP([Dict | Rest], Data, Name, Mod, Fmt, Opts, Code, Vid, Avp) -> +dec_AVP([Dict | Rest], Data, Name, Mod, Fmt, Opts0, Code, Vid, Avp) -> case Dict:avp_name(Code, Vid) of - {AvpName, Type} -> + {AvpName, Type} = NameT -> A = Avp#diameter_avp{name = AvpName, type = Type}, - #{failed_avp := Failed} = Opts, + #{failed_avp := Failed} + = Opts + = setopts(NameT, Name, Avp#diameter_avp.is_mandatory, Opts0), dec(Data, Name, AvpName, Type, Mod, Dict, Fmt, Failed, Opts, A); _ -> - dec_AVP(Rest, Data, Name, Mod, Fmt, Opts, Code, Vid, Avp) + dec_AVP(Rest, Data, Name, Mod, Fmt, Opts0, Code, Vid, Avp) end; dec_AVP([], _, _, _, _, _, _, _, Avp) -> @@ -680,30 +682,30 @@ set_failed('Failed-AVP', #{failed_avp := false} = Opts) -> set_failed(_, Opts) -> Opts. -%% acc/8 +%% acc/7 -acc([AM | Acc], As, I, Field, Arity, Strict, Mod, Opts) -> - [AM | acc1(Acc, As, I, Field, Arity, Strict, Mod, Opts)]. +acc([AM | Acc], As, I, Field, Arity, Strict, Mod) -> + [AM | acc1(Acc, As, I, Field, Arity, Strict, Mod)]. -%% acc1/8 +%% acc1/7 %% Faulty AVP, not grouped. -acc1(Acc, {_RC, Avp} = E, _, _, _, _, _, _) -> +acc1(Acc, {_RC, Avp} = E, _, _, _, _, _) -> [Avps, Failed | Rec] = Acc, [[Avp | Avps], [E | Failed] | Rec]; %% Faulty component in grouped AVP. -acc1(Acc, {RC, As, Avp}, _, _, _, _, _, _) -> +acc1(Acc, {RC, As, Avp}, _, _, _, _, _) -> [Avps, Failed | Rec] = Acc, [[As | Avps], [{RC, Avp} | Failed] | Rec]; %% Grouped AVP ... -acc1([Avps | Acc], [Avp|_] = As, I, Field, Arity, Strict, Mod, Opts) -> - [[As|Avps] | acc2(Acc, Avp, I, Field, Arity, Strict, Mod, Opts)]; +acc1([Avps | Acc], [Avp|_] = As, I, Field, Arity, Strict, Mod) -> + [[As|Avps] | acc2(Acc, Avp, I, Field, Arity, Strict, Mod)]; %% ... or not. -acc1([Avps | Acc], Avp, I, Field, Arity, Strict, Mod, Opts) -> - [[Avp|Avps] | acc2(Acc, Avp, I, Field, Arity, Strict, Mod, Opts)]. +acc1([Avps | Acc], Avp, I, Field, Arity, Strict, Mod) -> + [[Avp|Avps] | acc2(Acc, Avp, I, Field, Arity, Strict, Mod)]. %% The component list of a Grouped AVP is discarded when packing into %% the record (or equivalent): the values in an 'AVP' field are @@ -713,24 +715,24 @@ acc1([Avps | Acc], Avp, I, Field, Arity, Strict, Mod, Opts) -> %% retain the same structure as in diameter_packet.avps, but an 'AVP' %% list has always been flat. -%% acc2/8 +%% acc2/7 %% No errors, but nowhere to pack. -acc2(Acc, Avp, _, 'AVP', 0, _, _, _) -> +acc2(Acc, Avp, _, 'AVP', 0, _, _) -> [Failed | Rec] = Acc, [[{rc(Avp), Avp} | Failed] | Rec]; %% Relaxed arities. -acc2(Acc, Avp, _, Field, Arity, Strict, Mod, _) +acc2(Acc, Avp, _, Field, Arity, Strict, Mod) when Strict /= decode -> pack(Arity, Field, Avp, Mod, Acc); %% No maximum arity. -acc2(Acc, Avp, _, Field, {_,'*'} = Arity, _, Mod, _) -> +acc2(Acc, Avp, _, Field, {_,'*'} = Arity, _, Mod) -> pack(Arity, Field, Avp, Mod, Acc); %% Or check. -acc2(Acc, Avp, I, Field, Arity, _, Mod, _) -> +acc2(Acc, Avp, I, Field, Arity, _, Mod) -> Mx = max_arity(Arity), if Mx =< I -> [Failed | Rec] = Acc, diff --git a/lib/diameter/src/diameter.appup.src b/lib/diameter/src/diameter.appup.src index c2198de9ea..d0e58e8410 100644 --- a/lib/diameter/src/diameter.appup.src +++ b/lib/diameter/src/diameter.appup.src @@ -54,7 +54,9 @@ {"1.12.1", [{restart_application, diameter}]}, %% 19.1 {"1.12.2", [{restart_application, diameter}]}, %% 19.3 {"2.0", [{restart_application, diameter}]}, %% 20.0 - {"2.1", [{update, diameter_reg, {advanced, "2.1"}}]} %% 20.1 + {"2.1", [{load_module, diameter_gen}, %% 20.1 + {update, diameter_reg, {advanced, "2.1"}}]}, + {"2.1.1", [{load_module, diameter_gen}]} ], [ {"0.9", [{restart_application, diameter}]}, @@ -90,6 +92,7 @@ {"1.12.1", [{restart_application, diameter}]}, {"1.12.2", [{restart_application, diameter}]}, {"2.0", [{restart_application, diameter}]}, - {"2.1", [{restart_application, diameter}]} + {"2.1", [{restart_application, diameter}]}, + {"2.1.1", [{load_module, diameter_gen}]} ] }. diff --git a/lib/diameter/vsn.mk b/lib/diameter/vsn.mk index f73f68da0b..bfb260ed8f 100644 --- a/lib/diameter/vsn.mk +++ b/lib/diameter/vsn.mk @@ -17,5 +17,5 @@ # %CopyrightEnd% APPLICATION = diameter -DIAMETER_VSN = 2.1.1 +DIAMETER_VSN = 2.1.2 APP_VSN = $(APPLICATION)-$(DIAMETER_VSN)$(PRE_VSN) diff --git a/lib/inets/src/http_client/httpc_handler.erl b/lib/inets/src/http_client/httpc_handler.erl index bd1d2e833a..6907bf5262 100644 --- a/lib/inets/src/http_client/httpc_handler.erl +++ b/lib/inets/src/http_client/httpc_handler.erl @@ -736,7 +736,7 @@ maybe_send_answer(Request, Answer, State) -> answer_request(Request, Answer, State). deliver_answer(#request{from = From} = Request) - when is_pid(From) -> + when From =/= answer_sent -> Response = httpc_response:error(Request, socket_closed_remotely), httpc_response:send(From, Response); deliver_answer(_Request) -> diff --git a/lib/inets/src/http_server/httpd.erl b/lib/inets/src/http_server/httpd.erl index 0b632d24e3..540e68e749 100644 --- a/lib/inets/src/http_server/httpd.erl +++ b/lib/inets/src/http_server/httpd.erl @@ -99,7 +99,14 @@ start_service(Conf) -> stop_service({Address, Port}) -> stop_service({Address, Port, ?DEFAULT_PROFILE}); stop_service({Address, Port, Profile}) -> - httpd_sup:stop_child(Address, Port, Profile); + Name = httpd_util:make_name("httpd_instance_sup", Address, Port, Profile), + Pid = whereis(Name), + MonitorRef = erlang:monitor(process, Pid), + Result = httpd_sup:stop_child(Address, Port, Profile), + receive + {'DOWN', MonitorRef, _, _, _} -> + Result + end; stop_service(Pid) when is_pid(Pid) -> case service_info(Pid) of {ok, Info} -> diff --git a/lib/inets/src/http_server/httpd_request.erl b/lib/inets/src/http_server/httpd_request.erl index 0eaf073255..007d272323 100644 --- a/lib/inets/src/http_server/httpd_request.erl +++ b/lib/inets/src/http_server/httpd_request.erl @@ -306,10 +306,10 @@ add_chunk([<<>>, Body, Length, MaxChunk]) -> add_chunk([More, Body, Length, MaxChunk]) -> body_chunk(<<Body/binary, More/binary>>, Length, MaxChunk). -body_chunk(<<>> = Body, Length, MaxChunk) -> - {ok, {continue, ?MODULE, add_chunk, [Body, Length, MaxChunk]}}; body_chunk(Body, Length, nolimit) -> whole_body(Body, Length); +body_chunk(<<>> = Body, Length, MaxChunk) -> + {ok, {continue, ?MODULE, add_chunk, [Body, Length, MaxChunk]}}; body_chunk(Body, Length, MaxChunk) when Length > MaxChunk -> case size(Body) >= MaxChunk of diff --git a/lib/inets/src/http_server/httpd_request_handler.erl b/lib/inets/src/http_server/httpd_request_handler.erl index bd4fdd3832..d918f10424 100644 --- a/lib/inets/src/http_server/httpd_request_handler.erl +++ b/lib/inets/src/http_server/httpd_request_handler.erl @@ -516,6 +516,15 @@ handle_body(#state{headers = Headers, body = Body, case ((Length =< MaxBodySize) or (MaxBodySize == nolimit)) of true -> case httpd_request:body_chunk_first(Body, Length, MaxChunk) of + %% This is the case that the we need more data to complete + %% the body but chunking to the mod_esi user is not enabled. + {Module, add_chunk = Function, Args} -> + http_transport:setopts(ModData#mod.socket_type, + ModData#mod.socket, + [{active, once}]), + {noreply, State#state{mfa = + {Module, Function, Args}}}; + %% Chunking to mod_esi user is enabled {ok, {continue, Module, Function, Args}} -> http_transport:setopts(ModData#mod.socket_type, ModData#mod.socket, @@ -525,6 +534,8 @@ handle_body(#state{headers = Headers, body = Body, {ok, {{continue, Chunk}, Module, Function, Args}} -> handle_internal_chunk(State#state{chunk = chunk_start(MaxChunk), body = Chunk}, Module, Function, Args); + %% Whole body delivered, if chunking mechanism is enabled the whole + %% body fits in one chunk. {ok, NewBody} -> handle_response(State#state{chunk = chunk_finish(ChunkState, CbState, MaxChunk), diff --git a/lib/inets/test/httpc_SUITE.erl b/lib/inets/test/httpc_SUITE.erl index 5dfb1474e5..a39e786c79 100644 --- a/lib/inets/test/httpc_SUITE.erl +++ b/lib/inets/test/httpc_SUITE.erl @@ -130,7 +130,8 @@ only_simulated() -> port_in_host_header, redirect_port_in_host_header, relaxed, - multipart_chunks + multipart_chunks, + stream_fun_server_close ]. misc() -> @@ -745,7 +746,7 @@ empty_body() -> empty_body(Config) when is_list(Config) -> URL = url(group_name(Config), "/empty.html", Config), {ok, {{_,200,_}, [_ | _], []}} = - httpc:request(get, {URL, []}, [{timeout, 500}], []). + httpc:request(get, {URL, []}, [], []). %%------------------------------------------------------------------------- @@ -1178,6 +1179,22 @@ wait_for_whole_response(Config) when is_list(Config) -> ReqSeqNumServer ! shutdown. %%-------------------------------------------------------------------- +stream_fun_server_close() -> + [{doc, "Test that an error msg is received when using a receiver fun as stream target"}]. +stream_fun_server_close(Config) when is_list(Config) -> + Request = {url(group_name(Config), "/delay_close.html", Config), []}, + Self = self(), + Fun = fun(X) -> Self ! X end, + {ok, RequestId} = httpc:request(get, Request, [], [{sync, false}, {receiver, Fun}]), + receive + {RequestId, {error, Reason}} -> + ct:pal("Close ~p", [Reason]), + ok + after 13000 -> + ct:fail(did_not_receive_close) + end. + +%%-------------------------------------------------------------------- %% Internal Functions ------------------------------------------------ %%-------------------------------------------------------------------- stream(ReceiverPid, Receiver, Config) -> @@ -2029,6 +2046,9 @@ handle_uri(_,"/multipart_chunks.html",_,_,Socket,_) -> send(Socket, Head), send_multipart_chunks(Socket), http_chunk:encode_last(); +handle_uri(_,"/delay_close.html",_,_,Socket,_) -> + ct:sleep(10000), + close(Socket); handle_uri("HEAD",_,_,_,_,_) -> "HTTP/1.1 200 ok\r\n" ++ "Content-Length:0\r\n\r\n"; diff --git a/lib/inets/test/httpd_SUITE.erl b/lib/inets/test/httpd_SUITE.erl index 6c8728470b..0c649d9abf 100644 --- a/lib/inets/test/httpd_SUITE.erl +++ b/lib/inets/test/httpd_SUITE.erl @@ -129,7 +129,7 @@ groups() -> {http_1_1, [], [host, chunked, expect, cgi, cgi_chunked_encoding_test, trace, range, if_modified_since, mod_esi_chunk_timeout, - esi_put] ++ http_head() ++ http_get() ++ load()}, + esi_put, esi_post] ++ http_head() ++ http_get() ++ load()}, {http_1_0, [], [host, cgi, trace] ++ http_head() ++ http_get() ++ load()}, {http_0_9, [], http_head() ++ http_get() ++ load()} ]. @@ -932,7 +932,20 @@ esi_put() -> esi_put(Config) when is_list(Config) -> ok = http_status("PUT /cgi-bin/erl/httpd_example/put/123342234123 ", Config, [{statuscode, 200}]). - +%%------------------------------------------------------------------------- +esi_post() -> + [{doc, "Test mod_esi POST"}]. + +esi_post(Config) when is_list(Config) -> + Chunk = "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ", + Data = lists:duplicate(10000, Chunk), + Length = lists:flatlength(Data), + ok = http_status("POST /cgi-bin/erl/httpd_example/post ", + {"Content-Length:" ++ integer_to_list(Length) ++ "\r\n", + Data}, + [{http_version, "HTTP/1.1"} |Config], + [{statuscode, 200}]). + %%------------------------------------------------------------------------- mod_esi_chunk_timeout(Config) when is_list(Config) -> ok = httpd_1_1:mod_esi_chunk_timeout(proplists:get_value(type, Config), diff --git a/lib/inets/test/inets_SUITE.erl b/lib/inets/test/inets_SUITE.erl index 38b8229389..1abd96a228 100644 --- a/lib/inets/test/inets_SUITE.erl +++ b/lib/inets/test/inets_SUITE.erl @@ -213,7 +213,6 @@ start_httpd(Config) when is_list(Config) -> true = lists:member(Pid0, Pids0), [_|_] = inets:services_info(), inets:stop(httpd, Pid0), - ct:sleep(500), Pids1 = [ServicePid || {_, ServicePid} <- inets:services()], false = lists:member(Pid0, Pids1), {ok, Pid0b} = @@ -222,7 +221,6 @@ start_httpd(Config) when is_list(Config) -> true = lists:member(Pid0b, Pids0b), [_|_] = inets:services_info(), inets:stop(httpd, Pid0b), - ct:sleep(500), Pids1 = [ServicePid || {_, ServicePid} <- inets:services()], false = lists:member(Pid0b, Pids1), {ok, Pid1} = diff --git a/lib/kernel/test/zlib_SUITE.erl b/lib/kernel/test/zlib_SUITE.erl index 131a0685cd..7be7e503df 100644 --- a/lib/kernel/test/zlib_SUITE.erl +++ b/lib/kernel/test/zlib_SUITE.erl @@ -978,23 +978,37 @@ split_bin(Last,Acc) -> only_allow_owner(Config) when is_list(Config) -> Z = zlib:open(), + Owner = self(), ?m(ok, zlib:inflateInit(Z)), ?m(ok, zlib:inflateReset(Z)), {Pid, Ref} = spawn_monitor( fun() -> - ?m(?EXIT(not_on_controlling_process), zlib:inflateReset(Z)) + ?m(?EXIT(not_on_controlling_process), zlib:inflateReset(Z)), + Owner ! '$transfer_ownership', + receive + '$ownership_transferred' -> + ?m(ok, zlib:inflateReset(Z)) + after 200 -> + ct:fail("Never received transfer signal.") + end end), + ownership_transfer_check(Z, Pid, Ref). +ownership_transfer_check(Z, WorkerPid, Ref) -> receive - {'DOWN', Ref, process, Pid, _Reason} -> - ok + '$transfer_ownership' -> + zlib:set_controlling_process(Z, WorkerPid), + WorkerPid ! '$ownership_transferred', + ownership_transfer_check(Z, WorkerPid, Ref); + {'DOWN', Ref, process, WorkerPid, normal} -> + ok; + {'DOWN', Ref, process, WorkerPid, Reason} -> + ct:fail("Spawned worker crashed with reason ~p.", [Reason]) after 200 -> ct:fail("Spawned worker timed out.") - end, - - ?m(ok, zlib:inflateReset(Z)). + end. sub_heap_binaries(Config) when is_list(Config) -> Compressed = zlib:compress(<<"gurka">>), diff --git a/lib/parsetools/include/leexinc.hrl b/lib/parsetools/include/leexinc.hrl index b4449607cb..2a74c252ff 100644 --- a/lib/parsetools/include/leexinc.hrl +++ b/lib/parsetools/include/leexinc.hrl @@ -272,6 +272,8 @@ skip_cont(Rest, Line, {skip_token,Push}, Error) -> skip_cont(Rest, Line, {error,_S}, Error) -> skip_tokens(yystate(), Rest, Line, Rest, 0, Line, Error, reject, 0). +-compile({nowarn_unused_function, [yyrev/1, yyrev/2, yypre/2, yysuf/2]}). + yyrev(List) -> lists:reverse(List). yyrev(List, Tail) -> lists:reverse(List, Tail). yypre(List, N) -> lists:sublist(List, N). @@ -282,6 +284,8 @@ yysuf(List, N) -> lists:nthtail(N, List). %% Line has been updated with respect to newlines in the prefix of %% Chars consisting of (TokenLength - AcceptLength) characters. +-compile({nowarn_unused_function, adjust_line/4}). + adjust_line(N, N, _Cs, L) -> L; adjust_line(T, A, [$\n|Cs], L) -> adjust_line(T-1, A, Cs, L-1); diff --git a/lib/public_key/test/public_key_SUITE.erl b/lib/public_key/test/public_key_SUITE.erl index 579df160bc..38e8f30a25 100644 --- a/lib/public_key/test/public_key_SUITE.erl +++ b/lib/public_key/test/public_key_SUITE.erl @@ -97,14 +97,27 @@ end_per_group(_GroupName, Config) -> Config. %%------------------------------------------------------------------- -init_per_testcase(Case, Config) when Case == pkix_test_data_all_default; - Case == gen_ec_param -> +init_per_testcase(pkix_test_data_all_default, Config) -> case crypto:ec_curves() of [] -> {skip, missing_ecc_support}; _ -> - init_common_per_testcase(Config) + init_common_per_testcase(Config) end; + +init_per_testcase(gen_ec_param, Config) -> + case crypto:ec_curves() of + [] -> + {skip, missing_ecc_support}; + Curves -> + case lists:member(secp521r1, Curves) of + true -> + init_common_per_testcase(Config); + false -> + {skip, missing_ecc_secp52r1_support} + end + end; + init_per_testcase(TestCase, Config) -> case TestCase of ssh_hostkey_fingerprint_md5_implicit -> init_fingerprint_testcase([md5], Config); diff --git a/lib/sasl/src/format_lib_supp.erl b/lib/sasl/src/format_lib_supp.erl index cfe2ec7668..2d37dfe117 100644 --- a/lib/sasl/src/format_lib_supp.erl +++ b/lib/sasl/src/format_lib_supp.erl @@ -86,8 +86,10 @@ print_data(Device, Line, [{Key, Value}|T]) -> print_data(Device, Line, [Value|T]) -> Modifier = misc_supp:modifier(Device), io:format(Device, "~"++Modifier++"p~n", [Value]), - print_data(Device, Line, T). - + print_data(Device, Line, T); +print_data(Device, _Line, Value) -> + Modifier = misc_supp:modifier(Device), + io:format(Device, "~"++Modifier++"p~n", [Value]). print_items(Device, Line, {Name, Items}) -> print_items(Device, Line, Name, Items). diff --git a/lib/sasl/test/rb_SUITE.erl b/lib/sasl/test/rb_SUITE.erl index 426dedbab5..4ba2540545 100644 --- a/lib/sasl/test/rb_SUITE.erl +++ b/lib/sasl/test/rb_SUITE.erl @@ -36,6 +36,7 @@ no_group_cases() -> groups() -> [{running_error_logger,[shuffle],[show, + show_other, list, rescan, start_stop_log, @@ -165,6 +166,23 @@ show(Config) -> ok. +show_other(Config) -> + PrivDir = ?config(priv_dir,Config), + OutFile = filename:join(PrivDir,"rb_SUITE_log.txt"), + + %% Insert some reports in the error log and start rb + error_logger:info_report([rb_test_term_in_list]), + error_logger:info_report(rb_test_term_no_list), + ok = start_rb(OutFile), + + %% Show by type and check content + [{_,I1},{_,I2}] = check_report(fun() -> rb:show(info_report) end,OutFile), + + true = contains(I1,"rb_test_term_no_list"), + true = contains(I2,"rb_test_term_in_list"), + + ok. + list(Config) -> PrivDir = ?config(priv_dir,Config), OutFile = filename:join(PrivDir,"rb_SUITE_log.txt"), diff --git a/lib/snmp/doc/src/notes.xml b/lib/snmp/doc/src/notes.xml index 4705804759..6bdcae5dd7 100644 --- a/lib/snmp/doc/src/notes.xml +++ b/lib/snmp/doc/src/notes.xml @@ -34,7 +34,23 @@ </header> - <section><title>SNMP 5.2.7</title> + <section><title>SNMP 5.2.8</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The <c>recbuf</c> configuration option was not propagated + correctly to the socket for the SNMP Manager.</p> + <p> + Own Id: OTP-13372 Aux Id: ERIERL-73 </p> + </item> + </list> + </section> + +</section> + +<section><title>SNMP 5.2.7</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/snmp/src/app/snmp.appup.src b/lib/snmp/src/app/snmp.appup.src index bde637744c..ca61782639 100644 --- a/lib/snmp/src/app/snmp.appup.src +++ b/lib/snmp/src/app/snmp.appup.src @@ -8,19 +8,6 @@ %% {update, snmpa_local_db, soft, soft_purge, soft_purge, []} %% {add_module, snmpm_net_if_mt} [ - {<<"5\\.2\\.6">>, - [{load_module, snmpc, soft_purge, soft_purge, []}, - {load_module, snmpc_lib, soft_purge, soft_purge, []}]}, - {<<"5\\.2\\.5">>, - [{load_module, snmpc, soft_purge, soft_purge, []}, - {load_module, snmpc_lib, soft_purge, soft_purge, []}, - {load_module, snmp_generic, soft_purge, soft_purge, []}]}, - {<<"5\\.2\\.4">>, - [{load_module, snmpc, soft_purge, soft_purge, []}, - {load_module, snmp_generic, soft_purge, soft_purge, []}, - {load_module, snmp, soft_purge, soft_purge, []}, - {load_module, snmpc_lib, soft_purge, soft_purge, []}, - {load_module, snmpc_mib_gram, soft_purge, soft_purge, []}]}, {<<"5\\..*">>, [{restart_application, snmp}]}, {<<"4\\..*">>, [{restart_application, snmp}]} ], @@ -30,19 +17,6 @@ %% {remove, {snmpm_net_if_mt, soft_purge, soft_purge}} [ - {<<"5\\.2\\.6">>, - [{load_module, snmpc, soft_purge, soft_purge, []}, - {load_module, snmpc_lib, soft_purge, soft_purge, []}]}, - {<<"5\\.2\\.5">>, - [{load_module, snmpc, soft_purge, soft_purge, []}, - {load_module, snmpc_lib, soft_purge, soft_purge, []}, - {load_module, snmp_generic, soft_purge, soft_purge, []}]}, - {<<"5\\.2\\.4">>, - [{load_module, snmpc, soft_purge, soft_purge, []}, - {load_module, snmp_generic, soft_purge, soft_purge, []}, - {load_module, snmp, soft_purge, soft_purge, []}, - {load_module, snmpc_lib, soft_purge, soft_purge, []}, - {load_module, snmpc_mib_gram, soft_purge, soft_purge, []}]}, {<<"5\\..*">>, [{restart_application, snmp}]}, {<<"4\\..*">>, [{restart_application, snmp}]} ] diff --git a/lib/snmp/src/manager/snmpm_net_if.erl b/lib/snmp/src/manager/snmpm_net_if.erl index 93c987eb0f..29216f9d6a 100644 --- a/lib/snmp/src/manager/snmpm_net_if.erl +++ b/lib/snmp/src/manager/snmpm_net_if.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2004-2015. All Rights Reserved. +%% Copyright Ericsson AB 2004-2017. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -367,7 +367,7 @@ common_socket_opts(Opts) -> default -> []; Sz -> - [{sndbuf, Sz}] + [{recbuf, Sz}] end ++ case get_opt(Opts, no_reuse, false) of false -> diff --git a/lib/snmp/vsn.mk b/lib/snmp/vsn.mk index 207f0084d8..ef48608bda 100644 --- a/lib/snmp/vsn.mk +++ b/lib/snmp/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = snmp -SNMP_VSN = 5.2.7 +SNMP_VSN = 5.2.8 PRE_VSN = APP_VSN = "$(APPLICATION)-$(SNMP_VSN)$(PRE_VSN)" diff --git a/lib/stdlib/src/base64.erl b/lib/stdlib/src/base64.erl index 5885745fb1..c8cf6fdffe 100644 --- a/lib/stdlib/src/base64.erl +++ b/lib/stdlib/src/base64.erl @@ -113,9 +113,9 @@ encode_binary(Bin) -> Data :: ascii_binary(). decode(Bin) when is_binary(Bin) -> - decode_binary(<<>>, Bin); + decode_binary(Bin, <<>>); decode(List) when is_list(List) -> - list_to_binary(decode_l(List)). + decode_list(List, <<>>). -spec mime_decode(Base64) -> Data when Base64 :: ascii_string() | ascii_binary(), @@ -186,31 +186,41 @@ mime_decode_to_string(List) when is_list(List) -> bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad}). -decode_binary(Result0, <<C:8,T0/bits>>) -> - case element(C, ?DECODE_MAP) of - bad -> - erlang:error({badarg,C}); - ws -> - decode_binary(Result0, T0); - eq -> - case strip_ws(T0) of - <<$=:8,T/binary>> -> - <<>> = strip_ws(T), - Split = byte_size(Result0) - 1, - <<Result:Split/bytes,_:4>> = Result0, - Result; - T -> - <<>> = strip_ws(T), - Split = byte_size(Result0) - 1, - <<Result:Split/bytes,_:2>> = Result0, - Result - end; - Bits -> - decode_binary(<<Result0/bits,Bits:6>>, T0) +decode_binary(<<C1:8, Cs/bits>>, A) -> + case element(C1, ?DECODE_MAP) of + ws -> decode_binary(Cs, A); + B1 -> decode_binary(Cs, A, B1) end; -decode_binary(Result, <<>>) -> - true = is_binary(Result), - Result. +decode_binary(<<>>, A) -> + A. + +decode_binary(<<C2:8, Cs/bits>>, A, B1) -> + case element(C2, ?DECODE_MAP) of + ws -> decode_binary(Cs, A, B1); + B2 -> decode_binary(Cs, A, B1, B2) + end. + +decode_binary(<<C3:8, Cs/bits>>, A, B1, B2) -> + case element(C3, ?DECODE_MAP) of + ws -> decode_binary(Cs, A, B1, B2); + B3 -> decode_binary(Cs, A, B1, B2, B3) + end. + +decode_binary(<<C4:8, Cs/bits>>, A, B1, B2, B3) -> + case element(C4, ?DECODE_MAP) of + ws -> decode_binary(Cs, A, B1, B2, B3); + eq when B3 =:= eq -> only_ws_binary(Cs, <<A/binary,B1:6,(B2 bsr 4):2>>); + eq -> only_ws_binary(Cs, <<A/binary,B1:6,B2:6,(B3 bsr 2):4>>); + B4 -> decode_binary(Cs, <<A/binary,B1:6,B2:6,B3:6,B4:6>>) + end. + +only_ws_binary(<<>>, A) -> + A; +only_ws_binary(<<C:8, Cs/bits>>, A) -> + case element(C, ?DECODE_MAP) of + ws -> only_ws_binary(Cs, A); + _ -> erlang:error(function_clause) + end. %% Skipping pad character if not at end of string. Also liberal about %% excess padding and skipping of other illegal (non-base64 alphabet) @@ -262,6 +272,42 @@ mime_decode_binary_after_eq(Result0, <<>>, Eq) -> Result end. +decode_list([C1 | Cs], A) -> + case element(C1, ?DECODE_MAP) of + ws -> decode_list(Cs, A); + B1 -> decode_list(Cs, A, B1) + end; +decode_list([], A) -> + A. + +decode_list([C2 | Cs], A, B1) -> + case element(C2, ?DECODE_MAP) of + ws -> decode_list(Cs, A, B1); + B2 -> decode_list(Cs, A, B1, B2) + end. + +decode_list([C3 | Cs], A, B1, B2) -> + case element(C3, ?DECODE_MAP) of + ws -> decode_list(Cs, A, B1, B2); + B3 -> decode_list(Cs, A, B1, B2, B3) + end. + +decode_list([C4 | Cs], A, B1, B2, B3) -> + case element(C4, ?DECODE_MAP) of + ws -> decode_list(Cs, A, B1, B2, B3); + eq when B3 =:= eq -> only_ws(Cs, <<A/binary,B1:6,(B2 bsr 4):2>>); + eq -> only_ws(Cs, <<A/binary,B1:6,B2:6,(B3 bsr 2):4>>); + B4 -> decode_list(Cs, <<A/binary,B1:6,B2:6,B3:6,B4:6>>) + end. + +only_ws([], A) -> + A; +only_ws([C | Cs], A) -> + case element(C, ?DECODE_MAP) of + ws -> only_ws(Cs, A); + _ -> erlang:error(function_clause) + end. + decode([], A) -> A; decode([$=,$=,C2,C1|Cs], A) -> Bits2x6 = (b64d(C1) bsl 18) bor (b64d(C2) bsl 12), @@ -292,16 +338,6 @@ strip_spaces([$\r|Cs], A) -> strip_spaces(Cs, A); strip_spaces([$\n|Cs], A) -> strip_spaces(Cs, A); strip_spaces([C|Cs], A) -> strip_spaces(Cs, [C | A]). -strip_ws(<<$\t,T/binary>>) -> - strip_ws(T); -strip_ws(<<$\n,T/binary>>) -> - strip_ws(T); -strip_ws(<<$\r,T/binary>>) -> - strip_ws(T); -strip_ws(<<$\s,T/binary>>) -> - strip_ws(T); -strip_ws(T) -> T. - %% Skipping pad character if not at end of string. Also liberal about %% excess padding and skipping of other illegal (non-base64 alphabet) %% characters. See section 3.3 of RFC4648 diff --git a/otp_versions.table b/otp_versions.table index 46c160e63a..ba3bccfe67 100644 --- a/otp_versions.table +++ b/otp_versions.table @@ -1,3 +1,4 @@ +OTP-20.1.3 : diameter-2.1.2 erts-9.1.3 snmp-5.2.8 # asn1-5.0.3 common_test-1.15.2 compiler-7.1.3 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.2 cosTime-1.2.2 cosTransactions-1.3.2 crypto-4.1 debugger-4.2.3 dialyzer-3.2.2 edoc-0.9.1 eldap-1.2.2 erl_docgen-0.7.1 erl_interface-3.10 et-1.6.1 eunit-2.3.4 hipe-3.16.1 ic-4.4.2 inets-6.4.2 jinterface-1.8 kernel-5.4 megaco-3.18.2 mnesia-4.15.1 observer-2.5 odbc-2.12 orber-3.8.3 os_mon-2.4.3 otp_mibs-1.1.1 parsetools-2.1.5 public_key-1.5 reltool-0.7.5 runtime_tools-1.12.2 sasl-3.1 ssh-4.6.1 ssl-8.2.1 stdlib-3.4.2 syntax_tools-2.1.3 tools-2.11 wx-1.8.2 xmerl-1.3.15 : OTP-20.1.2 : diameter-2.1.1 erts-9.1.2 # asn1-5.0.3 common_test-1.15.2 compiler-7.1.3 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.2 cosTime-1.2.2 cosTransactions-1.3.2 crypto-4.1 debugger-4.2.3 dialyzer-3.2.2 edoc-0.9.1 eldap-1.2.2 erl_docgen-0.7.1 erl_interface-3.10 et-1.6.1 eunit-2.3.4 hipe-3.16.1 ic-4.4.2 inets-6.4.2 jinterface-1.8 kernel-5.4 megaco-3.18.2 mnesia-4.15.1 observer-2.5 odbc-2.12 orber-3.8.3 os_mon-2.4.3 otp_mibs-1.1.1 parsetools-2.1.5 public_key-1.5 reltool-0.7.5 runtime_tools-1.12.2 sasl-3.1 snmp-5.2.7 ssh-4.6.1 ssl-8.2.1 stdlib-3.4.2 syntax_tools-2.1.3 tools-2.11 wx-1.8.2 xmerl-1.3.15 : OTP-20.1.1 : compiler-7.1.3 erts-9.1.1 ssh-4.6.1 # asn1-5.0.3 common_test-1.15.2 cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.2 cosTime-1.2.2 cosTransactions-1.3.2 crypto-4.1 debugger-4.2.3 dialyzer-3.2.2 diameter-2.1 edoc-0.9.1 eldap-1.2.2 erl_docgen-0.7.1 erl_interface-3.10 et-1.6.1 eunit-2.3.4 hipe-3.16.1 ic-4.4.2 inets-6.4.2 jinterface-1.8 kernel-5.4 megaco-3.18.2 mnesia-4.15.1 observer-2.5 odbc-2.12 orber-3.8.3 os_mon-2.4.3 otp_mibs-1.1.1 parsetools-2.1.5 public_key-1.5 reltool-0.7.5 runtime_tools-1.12.2 sasl-3.1 snmp-5.2.7 ssl-8.2.1 stdlib-3.4.2 syntax_tools-2.1.3 tools-2.11 wx-1.8.2 xmerl-1.3.15 : OTP-20.1 : asn1-5.0.3 common_test-1.15.2 compiler-7.1.2 crypto-4.1 debugger-4.2.3 dialyzer-3.2.2 diameter-2.1 edoc-0.9.1 erl_docgen-0.7.1 erts-9.1 et-1.6.1 eunit-2.3.4 hipe-3.16.1 inets-6.4.2 kernel-5.4 mnesia-4.15.1 observer-2.5 os_mon-2.4.3 public_key-1.5 reltool-0.7.5 runtime_tools-1.12.2 sasl-3.1 snmp-5.2.7 ssh-4.6 ssl-8.2.1 stdlib-3.4.2 syntax_tools-2.1.3 tools-2.11 wx-1.8.2 # cosEvent-2.2.1 cosEventDomain-1.2.1 cosFileTransfer-1.2.1 cosNotification-1.2.2 cosProperty-1.2.2 cosTime-1.2.2 cosTransactions-1.3.2 eldap-1.2.2 erl_interface-3.10 ic-4.4.2 jinterface-1.8 megaco-3.18.2 odbc-2.12 orber-3.8.3 otp_mibs-1.1.1 parsetools-2.1.5 xmerl-1.3.15 : diff --git a/system/doc/oam/oam_intro.xml b/system/doc/oam/oam_intro.xml index d3867f03ca..ead8c026b9 100644 --- a/system/doc/oam/oam_intro.xml +++ b/system/doc/oam/oam_intro.xml @@ -211,7 +211,7 @@ snmp:c("MY-MIB", [{il, ["sasl/priv/mibs"]}]).</code> <p>The following MIBs are defined in the OTP system:</p> <list type="bulleted"> - <item><p><c>OTP-REG)</c> (in SASL) contains the top-level + <item><p><c>OTP-REG</c> (in SASL) contains the top-level OTP registration objects, used by all other MIBs.</p></item> <item><p><c>OTP-TC</c> (in SASL) contains the general Textual Conventions, which can be used by any other MIB.</p></item> |
