From 9af8bca495c1704d7ce010939c7e54df7bfdef71 Mon Sep 17 00:00:00 2001
From: Peter Andersson In order for the built-in handlers to survive, and stay responsive,
- during periods of high load (i.e. when huge numbers of incoming
- log requests must be handled), a mechanism for overload protection
- has been implemented in the
- The default handlers, The handler process keeps track of the length of its message
- queue and reacts in different ways depending on the current status.
- The purpose is to keep the handler in, or (as quickly as possible),
- get the handler into, a state where it can keep up with the pace
- of incoming log requests. The memory usage of the handler must never
- keep growing larger and larger, since that would eventually cause the
- handler to crash. Three thresholds with associated actions have been
- defined:
The default value of this level is
As long as the length of the message queue is lower than this
+ value, all log events are handled asynchronously. This means that
+ the process sending the log event, by calling a log function in the
+ Logger API, does not wait for a response from the handler, but
+ continues executing immediately after the event is sent (that is, it
+ is not affected by the time it takes the handler to print the event
+ to the log device). If the message queue grows larger than this value,
+ the handler starts handling log events synchronously instead,
+ meaning that the process sending the event, must wait for a
+ response. When the handler reduces the message queue to a
+ level below the
Default value of the threshold:
When the message queue has grown larger than this threshold, which
- defaults to
When the message queue grows larger than this threshold, the + handler switches to a mode in which it drops all new events that + senders want to send. Dropping an event in ths mode, means that the + log function never sends a message to the handler, but returns + without taking any action. The handler keeps logging events already + in the message queue, and when the length of the message queue is + reduced to a level below the threshold, synchronous or asynchronous + mode is resumed. Note that when the handler activates, or deactivates, + drop mode, information about it is printed to the log.
+Default value of the threshold:
Above this threshold, which defaults to
If the length of the message queue grows larger than this threshold, + a flush (delete) operation takes place. To flush events, the handler receives + the messages in the process mailbox in a loop without logging (that is, the + handler deletes the events). Processes waiting for a response from a + synchronous log request will receive a reply from the handler indicating + that the request has been dropped. The handler process will set its own priority + to high during the flush loop to make sure that no new events come in + during the operation. Note that after the flush operation is performed, + the handler prints information in the log about how many events have been + deleted
+Default value of the threshold:
For the overload protection algorithm to work properly, it is required that:
-and that:
-If
To disable certain modes, do the following:
+During high load scenarios, the length of the handler message queue rarely grows in a linear and predictable way. Instead, whenever the handler process gets scheduled in, it can have an almost arbitrary number of messages waiting in the mailbox. It's for this reason that the overload - protection mechanism is focused on acting quickly and quite drastically - (such as immediately dropping or flushing messages) as soon as a large - queue length is detected.
- -The thresholds listed above may be modified by the user if, e.g, a handler - shouldn't drop or flush messages unless the message queue length grows - extremely large. (The handler must be allowed to use large amounts of memory - under such circumstances however). Another example of when the user might want - to change the settings is if, for performance reasons, the logging processes must - never get blocked by synchronous log requests, while dropping or flushing requests - is perfectly acceptable (since it doesn't affect the performance of the - loggers).
+ protection mechanism is focused on acting quickly, and quite drastically, + such as immediately dropping or flushing messages, when a large queue length + is detected. + +The values of the previously listed thresholds can be specified by the user. + This way, a handler can be configured to, for example, not drop or flush + messages unless the message queue length of the handler process grows extremely + large. Note that large amounts of memory can be required for the node under such + circumstances. Another example of user configuration is when, for performance + reasons, the logging processes must never get blocked by synchronous log requests. + It's possible, perhaps, that dropping or flushing events is still acceptable (since + it does not affect the performance of the processes sending the log events).
A configuration example:
logger:add_handler(my_standard_h, logger_std_h,
#{config =>
#{type => {file,"./system_info.log"},
- toggle_sync_qlen => 100,
- drop_new_reqs_qlen => 1000,
- flush_reqs_qlen => 2000}}).
+ sync_mode_qlen => 100,
+ drop_mode_qlen => 1000,
+ flush_qlen => 2000}}).
A potential problem with large bursts of log requests, is that log files - may get full or wrapped too quickly (in the latter case overwriting - previously logged data that could be of great importance). For this reason, - both built-in handlers offer the possibility to set a maximum level of how - many requests to process with a certain time frame. With this burst control - feature enabled, the handler will take care of bursts of log requests - without choking log files, or the terminal, with massive amounts of - printouts. These are the configuration parameters:
+Large bursts of log events (that is, many events received by the handler + under a short period of time), can potentially cause problems, such as:
+For this reason, both built-in handlers offer the possibility to specify the + maximum number of events to be handled within a certain time frame. + With this burst control feature enabled, the handler can avoid choking the log with + massive amounts of printouts. These are the configuration parameters:
This is set to
Value
Defaults to
This is how many requests should be processed within the
-
This is the maximum number of events to handle within a
+
Defaults to
The default window is
See the previous description of
Defaults to
A configuration example:
logger:add_handler(my_disk_log_h, logger_disk_log_h,
- #{disk_log_opts =>
- #{file => "./my_disk_log"},
- config =>
- #{burst_limit_size => 10,
- burst_window_time => 500}}).
+ #{config =>
+ #{file => "./my_disk_log",
+ burst_limit_enable => true,
+ burst_limit_max_count => 20,
+ burst_limit_window_time => 500}}).
A handler process may grow large even if it can manage peaks of high load - without crashing. The overload protection mechanism includes user configurable - levels for a maximum allowed message queue length and maximum allowed memory - usage. This feature is disabled by default, but can be switched on by means - of the following configuration parameters:
- +It is possible that a handler, even if it can successfully manage peaks + of high load without crashing, can build up a large message queue, or use a + large amount of memory. The overload protection mechanism includes an + automatic termination and restart feature for the purpose of guaranteeing + that a handler does not grow out of bounds. The feature can be configured + with the following parameters:
This is set to
Value
Defaults to
This is the maximum allowed queue length. If the mailbox grows larger than this, the handler process gets terminated.
+Defaults to
This is the maximum allowed memory usage of the handler process. If - the handler grows any larger, the process gets terminated.
+This is the maximum memory size that the handler process is allowed to use. + If the handler grows larger than this, the process gets terminated.
+Defaults to
If the handler gets terminated because of its queue length or
- memory usage, it can get automatically restarted again after a
- configurable delay time. The time is specified in milliseconds
- and
If the handler gets terminated, it can restart automatically after a
+ delay, specified in milliseconds. The value
Defaults to
If the handler process gets terminated because of overload, information about + this event is printed in the log. Information about when a restart has taken + place, and the handler is back in action, is also printed.
This is a handler for Logger that offers circular
(wrapped) logs by using
The default standard handler,
The handler has an overload protection mechanism that will keep the handler - process and the Kernel application alive during a high load of log - requests. How this feature works, and how to modify the configuration, - is described in the +
The handler has an overload protection mechanism that keeps the handler
+ process and the Kernel application alive during high loads of log
+ events. How overload protection works, and how to configure it, is
+ described in the
To add a new instance of the disk_log handler, use
The settings for the disk_log log file should be specified with the
- key
Parameters in the
This is the full name of the disk_log log file. The option
+ corresponds to the
This is the disk_log type,
Defaults to
This is the maximum number of files that disk_log will use
+ for its circular logging. The option
+ corresponds to the
Defaults to
Specific configuration for the handler (represented as a sub map)
- is specified with the key
This is the maximum number of bytes that will be written to
+ a log file before disk_log proceeds with the next file in order (or
+ generates an error in case of a full halt log). The option
+ corresponds to the
Defaults to
This value (in milliseconds) specifies how often the handler will
- do a disk_log sync operation in order to make sure that buffered data
- gets written to disk. The handler will repeatedly attempt this
- operation, but only perform it if something has actually been logged
- since the last sync. The default value is
This value, in milliseconds, specifies how often the handler does + a disk_log sync operation to write buffered data to disk. The handler attempts + the operation repeatedly, but only performs a new sync if something has + actually been logged.
+Defaults to
If
There are a number of other configuration parameters available, that are - to be used for customizing the overload protection behaviour. The same - parameters are used both in the standard handler and the disk_log handler, - and are documented in the +
Other configuration parameters exist, to be used for customizing
+ the overload protection behaviour. The same parameters are used both in the
+ standard handler and the disk_log handler, and are documented in the
Note that when changing the configuration of the handler in runtime, by
calling
Example of adding a disk_log handler:
logger:add_handler(my_disk_log_h, logger_disk_log_h,
#{level => error,
filter_default => log,
- disk_log_opts =>
- #{file => "./my_disk_log",
- type => wrap,
- max_no_files => 4,
- max_no_bytes => 10000},
- config =>
- #{filesync_repeat_interval => 1000}}).
+ config => #{file => "./my_disk_log",
+ type => wrap,
+ max_no_files => 4,
+ max_no_bytes => 10000},
+ filesync_repeat_interval => 1000}}).
- In order to use the disk_log handler instead of the default standard +
To use the disk_log handler instead of the default standard handler when starting an Erlang node, change the Kernel default logger to use disk_log. Example:
erl -kernel logger '[{handler,default,logger_disk_log_h,
- #{disk_log_opts => #{file => "./system_disk_log"}}}]'
+ #{config => #{file => "./system_disk_log"}}}]'
Write buffered data to disk.
diff --git a/lib/kernel/doc/src/logger_std_h.xml b/lib/kernel/doc/src/logger_std_h.xml index 89e11389c5..08667ae7f9 100644 --- a/lib/kernel/doc/src/logger_std_h.xml +++ b/lib/kernel/doc/src/logger_std_h.xml @@ -38,62 +38,63 @@This is the default handler for Logger.
Multiple instances of this handler can be added to
- Logger, and each instance will print logs to
The handler has an overload protection mechanism that will keep the handler - process and the Kernel application alive during a high load of log - requests. How this feature works, and how to modify the configuration, - is described in the +
The handler has an overload protection mechanism that keeps the handler
+ process and the Kernel application alive during high loads of log
+ events. How overload protection works, and how to configure it, is
+ described in the
To add a new instance of the standard handler, use
This will have the value
This has the value
Defaults to
It is recommended to not specify
This value (in milliseconds) specifies how often the handler will
- do a file sync operation in order to make sure that buffered data gets
- written to disk. The handler will repeatedly attempt this
- operation, but only perform it if something has actually been logged
- since the last sync. The default value is
This value, in milliseconds, specifies how often the handler does + a file sync operation to write buffered data to disk. The handler attempts + the operation repeatedly, but only performs a new sync if something has + actually been logged.
+Defaults to
If
There are a number of other configuration parameters available, that are - to be used for customizing the overload protection behaviour. The same - parameters are used both in the standard handler and the disk_log handler, - and are documented in the +
Other configuration parameters exist, to be used for customizing
+ the overload protection behaviour. The same parameters are used both in the
+ standard handler and the disk_log handler, and are documented in the
Note that when changing the configuration of the handler in runtime, by +
Note that if changing the configuration of the handler in runtime, by
calling
Example of adding a standard handler:
logger:add_handler(my_standard_h, logger_std_h,
@@ -103,9 +104,9 @@ logger:add_handler(my_standard_h, logger_std_h,
#{type => {file,"./system_info.log"},
filesync_repeat_interval => 1000}}).
- In order to configure the default handler (that starts initially with +
To set the default handler (that starts initially with
the Kernel application) to log to file instead of
erl -kernel logger '[{handler,default,logger_std_h,
#{config => #{type => {file,"./log.log"}}}}]'
@@ -118,7 +119,7 @@ erl -kernel logger '[{handler,default,logger_std_h,
-
+
Writes buffered data to disk.
Write buffered data to disk.
--
cgit v1.2.3