From e146a3eec5a2d384260aa8829777c89eaab09cbd Mon Sep 17 00:00:00 2001 From: Lukas Larsson Date: Thu, 21 Apr 2016 18:36:45 +0200 Subject: erts: Implement max_heap_size process flag The max_heap_size process flag can be used to limit the growth of a process heap by killing it before it becomes too large to handle. It is possible to set the maximum using the `erl +hmax` option, `system_flag(max_heap_size, ...)`, `spawn_opt(Fun, [{max_heap_size, ...}])` and `process_flag(max_heap_size, ...)`. It is possible to configure the behaviour of the process when the maximum heap size is reached. The process may be sent an untrappable exit signal with reason kill and/or send an error_logger message with details on the process state. A new trace event called gc_max_heap_size is also triggered for the garbage_collection trace flag when the heap grows larger than the configured size. If kill and error_logger are disabled, it is still possible to see that the maximum has been reached by doing garbage collection tracing on the process. The heap size is defined as the sum of the heap memory that the process is currently using. This includes all generational heaps, the stack, any messages that are considered to be part of the heap and any extra memory the garbage collector may need during collection. In the current implementation this means that when a process is set using on_heap message queue data mode, the messages that are in the internal message queue are counted towards this value. For off_heap, only matched messages count towards the size of the heap. For mixed, it depends on race conditions within the VM whether a message is part of the heap or not. Below is an example run of the new behaviour: Eshell V8.0 (abort with ^G) 1> f(P),P = spawn_opt(fun() -> receive ok -> ok end end, [{max_heap_size, 512}]). <0.60.0> 2> erlang:trace(P, true, [garbage_collection, procs]). 1 3> [P ! lists:duplicate(M,M) || M <- lists:seq(1,15)],ok. ok 4> =ERROR REPORT==== 26-Apr-2016::16:25:10 === Process: <0.60.0> Context: maximum heap size reached Max heap size: 512 Total heap size: 723 Kill: true Error Logger: true GC Info: [{old_heap_block_size,0}, {heap_block_size,609}, {mbuf_size,145}, {recent_size,0}, {stack_size,9}, {old_heap_size,0}, {heap_size,211}, {bin_vheap_size,0}, {bin_vheap_block_size,46422}, {bin_old_vheap_size,0}, {bin_old_vheap_block_size,46422}] flush(). Shell got {trace,<0.60.0>,gc_start, [{old_heap_block_size,0}, {heap_block_size,233}, {mbuf_size,145}, {recent_size,0}, {stack_size,9}, {old_heap_size,0}, {heap_size,211}, {bin_vheap_size,0}, {bin_vheap_block_size,46422}, {bin_old_vheap_size,0}, {bin_old_vheap_block_size,46422}]} Shell got {trace,<0.60.0>,gc_max_heap_size, [{old_heap_block_size,0}, {heap_block_size,609}, {mbuf_size,145}, {recent_size,0}, {stack_size,9}, {old_heap_size,0}, {heap_size,211}, {bin_vheap_size,0}, {bin_vheap_block_size,46422}, {bin_old_vheap_size,0}, {bin_old_vheap_block_size,46422}]} Shell got {trace,<0.60.0>,exit,killed} --- erts/doc/src/erl.xml | 28 +++++ erts/doc/src/erlang.xml | 294 +++++++++++++++++++++++++++++++++++++----------- 2 files changed, 258 insertions(+), 64 deletions(-) (limited to 'erts/doc') diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml index c499fc8081..1bbde7f1e0 100644 --- a/erts/doc/src/erl.xml +++ b/erts/doc/src/erl.xml @@ -627,6 +627,34 @@

Sets the default binary virtual heap size of processes to the size .

+ + + +

Sets the default maximum heap size of processes to the size + . If +hmax is not given, the default is 0 + which means that no maximum heap size is used. + For more information, see the documentation of + + process_flag(max_heap_size, MaxHeapSize).

+ + + + +

Sets whether to send an error logger message for processes that reach + the maximum heap size or not. If +hmaxel is not given, the default is true. + For more information, see the documentation of + + process_flag(max_heap_size, MaxHeapSize).

+ + + + +

Sets whether to kill processes that reach the maximum heap size or not. If + +hmaxk is not given, the default is true. For more information, + see the documentation of + + process_flag(max_heap_size, MaxHeapSize).

+

Sets the initial process dictionary size of processes to the size diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index bf30cc7928..3930e5e11d 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -4322,6 +4322,7 @@ os_prompt% + Sets process flag min_heap_size for the calling process. @@ -4340,9 +4341,95 @@ os_prompt%

Returns the old value of the flag.

- + + + Sets process flag max_heap_size for the calling process. + +

+ This flag sets the maximum heap size for the calling process. + If MaxHeapSize is an integer, the system default + values for kill and error_logger are used. + + size + +

+ The maximum size in words of the process. If set to zero, the + heap size limit is disabled. Badarg will be thrown if the value is + smaller than + min_heap_size. + The size check is only done when a garbage collection is triggered. +

+

+ size is the entire heap of the process when garbage collection + is triggered, this includes all generational heaps, the process stack, + any + messages that are considered to be part of the heap and any + extra memory that the garbage collector needs during collection. +

+

+ size is the same as can be retrieved using + + erlang:process_info(Pid, total_heap_size), + or by adding heap_block_size, old_heap_block_size + and mbuf_size from + erlang:process_info(Pid, garbage_collection_info). +

+ + kill + +

+ When set to true the runtime system will send an + untrappable exit signal with reason kill to the process + if the maximum heap size is reached. The garbage collection + that triggered the kill will not be completed, instead the + process will exit as soon as is possible. When set to false + no exit signal will be sent to the process, instead it will + continue executing. +

+

+ If kill is not defined in the map + the system default will be used. The default system default + is true. It can be changed by either the erl + +hmaxk option, + or + erlang:system_flag(max_heap_size, MaxHeapSize). +

+ + error_logger + +

+ When set to true the runtime system will send a + message to the current error_logger + containing details about the process when the maximum + heap size is reached. One error_logger report will + be sent each time the limit is reached. +

+

+ If error_logger is not defined in the map the system + default will be used. The default system default is true. + It can be changed by either the erl +hmaxel + option, or + erlang:system_flag(max_heap_size, MaxHeapSize). +

+ +

+ The heap size of a process is quite hard to predict, especially the + amount of memory that is used during the garbage collection. When + contemplating using this option, it is recommended to first run + it in production with kill set to false and inspect + the error_logger reports to see what the normal peak sizes + of the processes in the system is and then tune the value + accordingly. +

+ +

+ + + + + Set process flag message_queue_data for the calling process @@ -4392,7 +4479,7 @@ os_prompt% - + Sets process flag priority for the calling process. @@ -4466,7 +4553,7 @@ os_prompt% - + Sets process flag save_calls for the calling process.

N must be an integer in the interval 0..10000. @@ -4497,7 +4584,7 @@ os_prompt% - + Sets process flag sensitive for the calling process.

Sets or clears flag sensitive for the current process. @@ -4551,6 +4638,7 @@ os_prompt% +

Returns a list containing InfoTuples with @@ -4604,6 +4692,7 @@ os_prompt% +

Returns information about the process identified by @@ -4696,6 +4785,7 @@ os_prompt% The content of GCInfo can be changed without prior notice.

+ {garbage_collection_info, GCInfo}

GCInfo is a list containing miscellaneous @@ -4875,6 +4965,7 @@ os_prompt% total suspend count on Suspendee, only the parts contributed by Pid.

 + {total_heap_size, Size}

Size is the total size, in words, of all heap @@ -5569,6 +5660,7 @@ true Creates a new process with a fun as entry point. + @@ -5586,6 +5678,7 @@ true Creates a new process with a fun as entry point on a given node. + @@ -5602,6 +5695,7 @@ true Creates a new process with a function as entry point. + @@ -5705,6 +5799,16 @@ true fine-tuning an application and to measure the execution time with various VSize values.

 + {max_heap_size, Size} + +

Sets the max_heap_size process flag. The default + max_heap_size is determined by the + +hmax erl + command line argument. For more information, see the + documentation of + process_flag(max_heap_size, + Size).

+ {message_queue_data, MQD}

Sets the state of the message_queue_data process @@ -5725,6 +5829,7 @@ true Creates a new process with a function as entry point on a given node. + @@ -6506,8 +6611,25 @@ ok + + + Sets system flag max_heap_size + +

+ Sets the default maximum heap size settings for processes. + The size is given in words. The new max_heap_size + effects only processes spawned efter the change has been made. + max_heap_size can be set for individual processes using + spawn_opt/N or + process_flag/2.

+

Returns the old value of the flag.

+ + + + + Sets system flag multi_scheduling.

@@ -6557,7 +6679,7 @@ ok - + Sets system flag scheduler_bind_type. @@ -6675,7 +6797,7 @@ ok - + Sets system flag scheduler_wall_time.

Turns on or off scheduler wall time measurements.

@@ -6685,7 +6807,7 @@ ok - + Sets system flag schedulers_online.

@@ -6710,7 +6832,7 @@ ok - + Sets system flag trace_control_word.

Sets the value of the node trace control word to @@ -6724,7 +6846,7 @@ ok - + Finalize the Time Offset

@@ -6990,6 +7112,81 @@ ok + + + + + + + + + + Information about the default process heap settings. + + + fullsweep_after + +

Returns {fullsweep_after, integer() >= 0}, which is + the fullsweep_after garbage collection setting used + by default. For more information, see + garbage_collection described in the following.

+ + garbage_collection + +

Returns a list describing the default garbage collection + settings. A process spawned on the local node by a + spawn or spawn_link uses these + garbage collection settings. The default settings can be + changed by using + system_flag/2. + spawn_opt/4 + can spawn a process that does not use the default + settings.

+ + max_heap_size + +

Returns {max_heap_size, MaxHeapSize}, + where MaxHeapSize is the current + system-wide max heap size settings for spawned processes. + This setting can be set using the erl command line + flags +hmax, + +hmaxk and + +hmaxel. It can + also be changed at run-time using + + erlang:system_flag(max_heap_size, MaxHeapSize). + For more details about the max_heap_size process flag + see + process_flag(max_heap_size, MaxHeapSize). +

+ + min_heap_size + +

Returns {min_heap_size, MinHeapSize}, + where MinHeapSize is the current + system-wide minimum heap size for spawned processes.

+ + message_queue_data + +

Returns the default value of the message_queue_data + process flag which is either off_heap, on_heap, or mixed. + This default is set by the erl command line argument + +hmqd. For more information on the + message_queue_data process flag, see documentation of + process_flag(message_queue_data, + MQD).

+ + min_bin_vheap_size + +

Returns {min_bin_vheap_size, + MinBinVHeapSize}, where + MinBinVHeapSize is the current system-wide + minimum binary virtual heap size for spawned processes.

+ + + + + @@ -7010,8 +7207,6 @@ ok - - @@ -7019,10 +7214,6 @@ ok - - - - @@ -7052,6 +7243,7 @@ ok + Information about the system.

Returns various information about the current system @@ -7288,25 +7480,6 @@ ok ERL_MAX_ETS_TABLES before starting the Erlang runtime system.

- fullsweep_after - -

Returns {fullsweep_after, integer() >= 0}, which is - the fullsweep_after garbage collection setting used - by default. For more information, see - garbage_collection described in the following.

- - garbage_collection - -

Returns a list describing the default garbage collection - settings. A process spawned on the local node by a - spawn or spawn_link uses these - garbage collection settings. The default settings can be - changed by using - system_flag/2. - spawn_opt/4 - can spawn a process that does not use the default - settings.

- heap_sizes

Returns a list of integers representing valid heap sizes @@ -7381,29 +7554,6 @@ ok

Returns a string containing the Erlang machine name.

 - min_heap_size - -

Returns {min_heap_size, MinHeapSize}, - where MinHeapSize is the current - system-wide minimum heap size for spawned processes.

- - message_queue_data - -

Returns the default value of the message_queue_data - process flag which is either off_heap, on_heap, or mixed. - This default is set by the erl command line argument - +hmqd. For more information on the - message_queue_data process flag, see documentation of - process_flag(message_queue_data, - MQD).

- - min_bin_vheap_size - -

Returns {min_bin_vheap_size, - MinBinVHeapSize}, where - MinBinVHeapSize is the current system-wide - minimum binary virtual heap size for spawned processes.

- modified_timing_level

Returns the modified timing-level (an integer) if @@ -8028,12 +8178,13 @@ ok GcPid and Info are the same as for long_gc earlier, except that the tuple tagged with timeout is not present.

-

As of ERTS 5.6, the monitor message is sent - if the sum of the sizes of all memory blocks allocated - for all heap generations is equal to or higher than Size. - Previously the monitor message was sent if the memory block - allocated for the youngest generation was equal to or higher - than Size.

+

The monitor message is sent if the sum of the sizes of + all memory blocks allocated for all heap generations after + a garbage collection is equal to or higher than Size.

+

When a process is killed by + max_heap_size, it is killed before the + garbage collection is complete and thus no large heap message + will be sent.

 busy_port @@ -8560,7 +8711,9 @@ timestamp() -> garbage_collection

Traces garbage collections of processes.

-

Message tags: gc_minor_start and gc_minor_end.

+

Message tags: gc_minor_start, + gc_max_heap_size and + gc_minor_end.

 timestamp @@ -8928,6 +9081,19 @@ timestamp() ->

All sizes are in words.

 + + + {trace, Pid, gc_max_heap_size, Info} + + +

+ Sent when the max_heap_size + is reached during garbage collection. Info contains the + same kind of list as in message gc_start, + but the sizes reflect the sizes that triggered max_heap_size to + be reached. +

+ {trace, Pid, gc_minor_end, Info} -- cgit v1.2.3