aboutsummaryrefslogtreecommitdiffstats
path: root/erts/doc/src/crash_dump.xml
diff options
context:
space:
mode:
authorErlang/OTP <[email protected]>2009-11-20 14:54:40 +0000
committerErlang/OTP <[email protected]>2009-11-20 14:54:40 +0000
commit84adefa331c4159d432d22840663c38f155cd4c1 (patch)
treebff9a9c66adda4df2106dfd0e5c053ab182a12bd /erts/doc/src/crash_dump.xml
downloadotp-84adefa331c4159d432d22840663c38f155cd4c1.tar.gz
otp-84adefa331c4159d432d22840663c38f155cd4c1.tar.bz2
otp-84adefa331c4159d432d22840663c38f155cd4c1.zip
The R13B03 release.OTP_R13B03
Diffstat (limited to 'erts/doc/src/crash_dump.xml')
-rw-r--r--erts/doc/src/crash_dump.xml518
1 files changed, 518 insertions, 0 deletions
diff --git a/erts/doc/src/crash_dump.xml b/erts/doc/src/crash_dump.xml
new file mode 100644
index 0000000000..5182929358
--- /dev/null
+++ b/erts/doc/src/crash_dump.xml
@@ -0,0 +1,518 @@
+<?xml version="1.0" encoding="latin1" ?>
+<!DOCTYPE chapter SYSTEM "chapter.dtd">
+
+<chapter>
+ <header>
+ <copyright>
+ <year>1999</year><year>2009</year>
+ <holder>Ericsson AB. All Rights Reserved.</holder>
+ </copyright>
+ <legalnotice>
+ The contents of this file are subject to the Erlang Public License,
+ Version 1.1, (the "License"); you may not use this file except in
+ compliance with the License. You should have received a copy of the
+ Erlang Public License along with this software. If not, it can be
+ retrieved online at http://www.erlang.org/.
+
+ Software distributed under the License is distributed on an "AS IS"
+ basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+ the License for the specific language governing rights and limitations
+ under the License.
+
+ </legalnotice>
+
+ <title>How to interpret the Erlang crash dumps</title>
+ <prepared>Patrik Nyblom</prepared>
+ <responsible></responsible>
+ <docno></docno>
+ <approved></approved>
+ <checked></checked>
+ <date>1999-11-11</date>
+ <rev>PA1</rev>
+ <file>crash_dump.xml</file>
+ </header>
+ <p>This document describes the <c><![CDATA[erl_crash.dump]]></c> file generated
+ upon abnormal exit of the Erlang runtime system.</p>
+ <p><em>Important:</em> For OTP release R9C the Erlang crash dump has
+ had a major facelift. This means that the information in this
+ document will not be directly applicable for older dumps. However,
+ if you use the Crashdump Viewer tool on older dumps, the crash
+ dumps are translated into a format similar to this.</p>
+ <p>The system will write the crash dump in the current directory of
+ the emulator or in the file pointed out by the environment variable
+ (whatever that means on the current operating system)
+ ERL_CRASH_DUMP. For a crash dump to be written, there has to be a
+ writable file system mounted.</p>
+ <p>Crash dumps are written mainly for one of two reasons: either the
+ builtin function <c><![CDATA[erlang:halt/1]]></c> is called explicitly with a
+ string argument from running Erlang code, or else the runtime
+ system has detected an error that cannot be handled. The most
+ usual reason that the system can't handle the error is that the
+ cause is external limitations, such as running out of memory. A
+ crash dump due to an internal error may be caused by the system
+ reaching limits in the emulator itself (like the number of atoms
+ in the system, or too many simultaneous ets tables). Usually the
+ emulator or the operating system can be reconfigured to avoid the
+ crash, which is why interpreting the crash dump correctly is
+ important.</p>
+ <p>The erlang crash dump is a readable text file, but it might not be
+ very easy to read. Using the Crashdump Viewer tool in the
+ <c><![CDATA[observer]]></c> application will simplify the task. This is an
+ HTML based tool for browsing Erlang crash dumps.</p>
+
+ <section>
+ <marker id="general_info"></marker>
+ <title>General information</title>
+ <p>The first part of the dump shows the creation time for the dump,
+ a slogan indicating the reason for the dump, the system version,
+ of the node from which the dump originates, the compile time of
+ the emulator running the originating node and the number of
+ atoms in the atom table.
+ </p>
+
+ <section>
+ <title>Reasons for crash dumps (slogan)</title>
+ <p>The reason for the dump is noted in the beginning of the file
+ as <em>Slogan: &lt;reason&gt;</em> (the word "slogan" has historical
+ roots). If the system is halted by the BIF
+ <c><![CDATA[erlang:halt/1]]></c>, the slogan is the string parameter
+ passed to the BIF, otherwise it is a description generated by
+ the emulator or the (Erlang) kernel. Normally the message
+ should be enough to understand the problem, but nevertheless
+ some messages are described here. Note however that the
+ suggested reasons for the crash are <em>only suggestions</em>. The exact reasons for the errors may vary
+ depending on the local applications and the underlying
+ operating system.</p>
+ <list type="bulleted">
+ <item>"<em>&lt;A&gt;</em>: Cannot allocate <em>&lt;N&gt;</em>
+ bytes of memory (of type "<em>&lt;T&gt;</em>")." - The system
+ has run out of memory. &lt;A&gt; is the allocator that failed
+ to allocate memory, &lt;N&gt; is the number of bytes that
+ &lt;A&gt; tried to allocate, and &lt;T&gt; is the memory block
+ type that the memory was needed for. The most common case is
+ that a process stores huge amounts of data. In this case
+ &lt;T&gt; is most often <c><![CDATA[heap]]></c>, <c><![CDATA[old_heap]]></c>,
+ <c><![CDATA[heap_frag]]></c>, or <c><![CDATA[binary]]></c>. For more information on
+ allocators see
+ <seealso marker="erts_alloc">erts_alloc(3)</seealso>.</item>
+ <item>"<em>&lt;A&gt;</em>: Cannot reallocate <em>&lt;N&gt;</em>
+ bytes of memory\011(of type "<em>&lt;T&gt;</em>")." - Same as
+ above with the exception that memory was being reallocated
+ instead of being allocated when the system ran out of memory.</item>
+ <item>"Unexpected op code <em>N</em>" - Error in compiled
+ code, <c><![CDATA[beam]]></c> file damaged or error in the compiler.</item>
+ <item>"Module <em>Name</em> undefined" <c><![CDATA[|]]></c> "Function
+ <em>Name</em> undefined" <c><![CDATA[|]]></c> "No function
+ <em>Name</em>:<em>Name</em>/1" <c><![CDATA[|]]></c> "No function
+ <em>Name</em>:start/2" - The kernel/stdlib applications are
+ damaged or the start script is damaged.</item>
+ <item>"Driver_select called with too large file descriptor
+ <c><![CDATA[N]]></c>" - The number of file descriptors for sockets
+ exceed 1024 (Unix only). The limit on file-descriptors in
+ some Unix flavors can be set to over 1024, but only 1024
+ sockets/pipes can be used simultaneously by Erlang (due to
+ limitations in the Unix <c><![CDATA[select]]></c> call). The number of
+ open regular files is not affected by this.</item>
+ <item>"Received SIGUSR1" - The SIGUSR1 signal was sent to the
+ Erlang machine (Unix only).</item>
+ <item>"Kernel pid terminated (<em>Who</em>)
+ (<em>Exit-reason</em>)" - The kernel supervisor has detected
+ a failure, usually that the <c><![CDATA[application_controller]]></c>
+ has shut down (<c><![CDATA[Who]]></c> = <c><![CDATA[application_controller]]></c>,
+ <c><![CDATA[Why]]></c> = <c><![CDATA[shutdown]]></c>). The application controller
+ may have shut down for a number of reasons, the most usual
+ being that the node name of the distributed Erlang node is
+ already in use. A complete supervisor tree "crash" (i.e.,
+ the top supervisors have exited) will give about the same
+ result. This message comes from the Erlang code and not from
+ the virtual machine itself. It is always due to some kind of
+ failure in an application, either within OTP or a
+ "user-written" one. Looking at the error log for your
+ application is probably the first step to take.</item>
+ <item>"Init terminating in do_boot ()" - The primitive Erlang boot
+ sequence was terminated, most probably because the boot
+ script has errors or cannot be read. This is usually a
+ configuration error - the system may have been started with
+ a faulty <c><![CDATA[-boot]]></c> parameter or with a boot script from
+ the wrong version of OTP.</item>
+ <item>"Could not start kernel pid (<em>Who</em>) ()" - One of the
+ kernel processes could not start. This is probably due to
+ faulty arguments (like errors in a <c><![CDATA[-config]]></c> argument)
+ or faulty configuration files. Check that all files are in
+ their correct location and that the configuration files (if
+ any) are not damaged. Usually there are also messages
+ written to the controlling terminal and/or the error log
+ explaining what's wrong.</item>
+ </list>
+ <p>Other errors than the ones mentioned above may occur, as the
+ <c><![CDATA[erlang:halt/1]]></c> BIF may generate any message. If the
+ message is not generated by the BIF and does not occur in the
+ list above, it may be due to an error in the emulator. There
+ may however be unusual messages that I haven't mentioned, that
+ still are connected to an application failure. There is a lot
+ more information available, so more thorough reading of the
+ crash dump may reveal the crash reason. The size of processes,
+ the number of ets tables and the Erlang data on each process
+ stack can be useful for tracking down the problem.</p>
+ </section>
+
+ <section>
+ <title>Number of atoms</title>
+ <p>The number of atoms in the system at the time of the crash is
+ shown as <em>Atoms: &lt;number&gt;</em>. Some ten thousands atoms is
+ perfectly normal, but more could indicate that the BIF
+ <c><![CDATA[erlang:list_to_atom/1]]></c> is used to dynamically generate a
+ lot of <em>different</em> atoms, which is never a good idea.</p>
+ </section>
+ </section>
+
+ <section>
+ <marker id="memory"></marker>
+ <title>Memory information</title>
+ <p>Under the tag <em>=memory</em> you will find information similar
+ to what you can obtain on a living node with
+ <seealso marker="erts:erlang#erlang:memory/0">erlang:memory()</seealso>.</p>
+ </section>
+
+ <section>
+ <marker id="internal_tables"></marker>
+ <title>Internal table information</title>
+ <p>The tags <em>=hash_table:&lt;table_name&gt;</em> and
+ <em>=index_table:&lt;table_name&gt;</em> presents internal
+ tables. These are mostly of interest for runtime system
+ developers.</p>
+ </section>
+
+ <section>
+ <marker id="allocated_areas"></marker>
+ <title>Allocated areas</title>
+ <p>Under the tag <em>=allocated_areas</em> you will find information
+ similar to what you can obtain on a living node with
+ <seealso marker="erts:erlang#system_info_allocated_areas">erlang:system_info(allocated_areas)</seealso>.</p>
+ </section>
+
+ <section>
+ <marker id="allocator"></marker>
+ <title>Allocator</title>
+ <p>Under the tag <em>=allocator:&lt;A&gt;</em> you will find
+ various information about allocator &lt;A&gt;. The information
+ is similar to what you can obtain on a living node with
+ <seealso marker="erts:erlang#system_info_allocator_tuple">erlang:system_info({allocator, &lt;A&gt;})</seealso>.
+ For more information see the documentation of
+ <seealso marker="erts:erlang#system_info_allocator_tuple">erlang:system_info({allocator, &lt;A&gt;})</seealso>,
+ and the
+ <seealso marker="erts_alloc">erts_alloc(3)</seealso>
+ documentation.</p>
+ </section>
+
+ <section>
+ <marker id="processes"></marker>
+ <title>Process information</title>
+ <p>The Erlang crashdump contains a listing of each living Erlang
+ process in the system. The process information for one process
+ may look like this (line numbers have been added):
+ </p>
+ <p>The following fields can exist for a process:</p>
+ <taglist>
+ <tag><em>=proc:&lt;pid&gt;</em></tag>
+ <item>Heading, states the process identifier</item>
+ <tag><em>State</em></tag>
+ <item>
+ <p>The state of the process. This can be one of the following:</p>
+ <list type="bulleted">
+ <item><em>Scheduled</em> - The process was scheduled to run
+ but not currently running ("in the run queue").</item>
+ <item><em>Waiting</em> - The process was waiting for
+ something (in <c><![CDATA[receive]]></c>).</item>
+ <item><em>Running</em> - The process was currently
+ running. If the BIF <c><![CDATA[erlang:halt/1]]></c> was called, this was
+ the process calling it.</item>
+ <item><em>Exiting</em> - The process was on its way to
+ exit.</item>
+ <item><em>Garbing</em> - This is bad luck, the process was
+ garbage collecting when the crash dump was written, the rest
+ of the information for this process is limited.</item>
+ <item><em>Suspended</em> - The process is suspended, either
+ by the BIF <c><![CDATA[erlang:suspend_process/1]]></c> or because it is
+ trying to write to a busy port.</item>
+ </list>
+ </item>
+ <tag><em>Registered name</em></tag>
+ <item>The registered name of the process, if any.</item>
+ <tag><em>Spawned as</em></tag>
+ <item>The entry point of the process, i.e., what function was
+ referenced in the <c><![CDATA[spawn]]></c> or <c><![CDATA[spawn_link]]></c> call that
+ started the process.</item>
+ <tag><em>Last scheduled in for | Current call</em></tag>
+ <item>The current function of the process. These fields will not
+ always exist.</item>
+ <tag><em>Spawned by</em></tag>
+ <item>The parent of the process, i.e. the process which executed
+ <c><![CDATA[spawn]]></c> or <c><![CDATA[spawn_link]]></c>.</item>
+ <tag><em>Started</em></tag>
+ <item>The date and time when the process was started.</item>
+ <tag><em>Message queue length</em></tag>
+ <item>The number of messages in the process' message queue.</item>
+ <tag><em>Number of heap fragments</em></tag>
+ <item>The number of allocated heap fragments.</item>
+ <tag><em>Heap fragment data</em></tag>
+ <item>Size of fragmented heap data. This is data either created by
+ messages being sent to the process or by the Erlang BIFs. This
+ amount depends on so many things that this field is utterly
+ uninteresting.</item>
+ <tag><em>Link list</em></tag>
+ <item>Process id's of processes linked to this one. May also contain
+ ports. If process monitoring is used, this field also tells in
+ which direction the monitoring is in effect, i.e., a link
+ being "to" a process tells you that the "current" process was
+ monitoring the other and a link "from" a process tells you
+ that the other process was monitoring the current one.</item>
+ <tag><em>Reductions</em></tag>
+ <item>The number of reductions consumed by the process.</item>
+ <tag><em>Stack+heap</em></tag>
+ <item>The size of the stack and heap (they share memory segment)</item>
+ <tag><em>OldHeap</em></tag>
+ <item>The size of the "old heap". The Erlang virtual machine uses
+ generational garbage collection with two generations. There is
+ one heap for new data items and one for the data that have
+ survived two garbage collections. The assumption (which is
+ almost always correct) is that data that survive two garbage
+ collections can be "tenured" to a heap more seldom garbage
+ collected, as they will live for a long period. This is a
+ quite usual technique in virtual machines. The sum of the
+ heaps and stack together constitute most of the process's
+ allocated memory.</item>
+ <tag><em>Heap unused, OldHeap unused</em></tag>
+ <item>The amount of unused memory on each heap. This information is
+ usually useless.</item>
+ <tag><em>Stack</em></tag>
+ <item>If the system uses shared heap, the fields
+ <em>Stack+heap</em>, <em>OldHeap</em>, <em>Heap unused</em>
+ and <em>OldHeap unused</em> do not exist. Instead this field
+ presents the size of the process' stack.</item>
+ <tag><em>Program counter</em></tag>
+ <item>The current instruction pointer. This is only interesting for
+ runtime system developers. The function into which the program
+ counter points is the current function of the process.</item>
+ <tag><em>CP</em></tag>
+ <item>The continuation pointer, i.e. the return address for the
+ current call. Usually useless for other than runtime system
+ developers. This may be followed by the function into which
+ the CP points, which is the function calling the current
+ function.</item>
+ <tag><em>Arity</em></tag>
+ <item>The number of live argument registers. The argument registers,
+ if any are live, will follow. These may contain the arguments
+ of the function if they are not yet moved to the stack.</item>
+ </taglist>
+ <p>See also the section about <seealso marker="#proc_data">process data</seealso>.</p>
+ </section>
+
+ <section>
+ <marker id="ports"></marker>
+ <title>Port information</title>
+ <p>This section lists the open ports, their owners, any linked
+ processed, and the name of their driver or external process.</p>
+ </section>
+
+ <section>
+ <marker id="ets_tables"></marker>
+ <title>ETS tables</title>
+ <p>This section contains information about all the ETS tables in
+ the system. The following fields are interesting for each table:</p>
+ <taglist>
+ <tag><em>=ets:&lt;owner&gt;</em></tag>
+ <item>Heading, states the owner of the table (a process identifier)</item>
+ <tag><em>Table</em></tag>
+ <item>The identifier for the table. If the table is a
+ <c><![CDATA[named_table]]></c>, this is the name.</item>
+ <tag><em>Name</em></tag>
+ <item>The name of the table, regardless of whether it is a
+ <c><![CDATA[named_table]]></c> or not.</item>
+ <tag><em>Buckets</em></tag>
+ <item>This occurs if the table is a hash table, i.e. if it is not an
+ <c><![CDATA[ordered_set]]></c>.</item>
+ <tag><em>Ordered set (AVL tree), Elements</em></tag>
+ <item>This occurs only if the table is an <c><![CDATA[ordered_set]]></c>. (The
+ number of elements is the same as the number of objects in the
+ table.)</item>
+ <tag><em>Objects</em></tag>
+ <item>The number of objects in the table</item>
+ <tag><em>Words</em></tag>
+ <item>The number of words (usually 4 bytes/word) allocated to data
+ in the table.</item>
+ </taglist>
+ </section>
+
+ <section>
+ <marker id="timers"></marker>
+ <title>Timers</title>
+ <p>This section contains information about all the timers started
+ with the BIFs <c><![CDATA[erlang:start_timer/3]]></c> and
+ <c><![CDATA[erlang:send_after/3]]></c>. The following fields exists for each
+ timer:</p>
+ <taglist>
+ <tag><em>=timer:&lt;owner&gt;</em></tag>
+ <item>Heading, states the owner of the timer (a process identifier)
+ i.e. the process to receive the message when the timer
+ expires.</item>
+ <tag><em>Message</em></tag>
+ <item>The message to be sent.</item>
+ <tag><em>Time left</em></tag>
+ <item>Number of milliseconds left until the message would have been
+ sent.</item>
+ </taglist>
+ </section>
+
+ <section>
+ <marker id="distribution_info"></marker>
+ <title>Distribution information</title>
+ <p>If the Erlang node was alive, i.e., set up for communicating
+ with other nodes, this section lists the connections that were
+ active. The following fields can exist:</p>
+ <taglist>
+ <tag><em>=node:&lt;node_name&gt;</em></tag>
+ <item>The name of the node</item>
+ <tag><em>no_distribution</em></tag>
+ <item>This will only occur if the node was not distributed.</item>
+ <tag><em>=visible_node:&lt;channel&gt;</em></tag>
+ <item>Heading for a visible nodes, i.e. an alive node with a
+ connection to the node that crashed. States the channel number
+ for the node.</item>
+ <tag><em>=hidden_node:&lt;channel&gt;</em></tag>
+ <item>Heading for a hidden node. A hidden node is the same as a
+ visible node, except that it is started with the "-hidden"
+ flag. States the channel number for the node.</item>
+ <tag><em>=not_connected:&lt;channel&gt;</em></tag>
+ <item>Heading for a node which is has been connected to the crashed
+ node earlier. References (i.e. process or port identifiers)
+ to the not connected node existed at the time of the crash.
+ exist. States the channel number for the node.</item>
+ <tag><em>Name</em></tag>
+ <item>The name of the remote node.</item>
+ <tag><em>Controller</em></tag>
+ <item>The port which controls the communication with the remote node.</item>
+ <tag><em>Creation</em></tag>
+ <item>An integer (1-3) which together with the node name identifies
+ a specific instance of the node.</item>
+ <tag><em>Remote monitoring: &lt;local_proc&gt; &lt;remote_proc&gt;</em></tag>
+ <item>The local process was monitoring the remote process at the
+ time of the crash.</item>
+ <tag><em>Remotely monitored by: &lt;local_proc&gt; &lt;remote_proc&gt;</em></tag>
+ <item>The remote process was monitoring the local process at the
+ time of the crash.</item>
+ <tag><em>Remote link: &lt;local_proc&gt; &lt;remote_proc&gt;</em></tag>
+ <item>A link existed between the local process and the remote
+ process at the time of the crash.</item>
+ </taglist>
+ </section>
+
+ <section>
+ <marker id="loaded_modules"></marker>
+ <title>Loaded module information</title>
+ <p>This section contains information about all loaded modules.
+ First, the memory usage by loaded code is summarized. There is
+ one field for "Current code" which is code that is the current
+ latest version of the modules. There is also a field for "Old
+ code" which is code where there exists a newer version in the
+ system, but the old version is not yet purged. The memory usage
+ is in bytes.</p>
+ <p>All loaded modules are then listed. The following fields exist:</p>
+ <taglist>
+ <tag><em>=mod:&lt;module_name&gt;</em></tag>
+ <item>Heading, and the name of the module.</item>
+ <tag><em>Current size</em></tag>
+ <item>Memory usage for the loaded code in bytes</item>
+ <tag><em>Old size</em></tag>
+ <item>Memory usage for the old code, if any.</item>
+ <tag><em>Current attributes</em></tag>
+ <item>Module attributes for the current code. This field is decoded
+ when looked at by the Crashdump Viewer tool.</item>
+ <tag><em>Old attributes</em></tag>
+ <item>Module attributes for the old code, if any. This field is
+ decoded when looked at by the Crashdump Viewer tool.</item>
+ <tag><em>Current compilation info</em></tag>
+ <item>Compilation information (options) for the current code. This
+ field is decoded when looked at by the Crashdump Viewer tool.</item>
+ <tag><em>Old compilation info</em></tag>
+ <item>Compilation information (options) for the old code, if
+ any. This field is decoded when looked at by the Crashdump
+ Viewer tool.</item>
+ </taglist>
+ </section>
+
+ <section>
+ <marker id="funs"></marker>
+ <title>Fun information</title>
+ <p>In this section, all funs are listed. The following fields exist
+ for each fun:</p>
+ <taglist>
+ <tag><em>=fun</em></tag>
+ <item>Heading</item>
+ <tag><em>Module</em></tag>
+ <item>The name of the module where the fun was defined.</item>
+ <tag><em>Uniq, Index</em></tag>
+ <item>Identifiers</item>
+ <tag><em>Address</em></tag>
+ <item>The address of the fun's code.</item>
+ <tag><em>Native_address</em></tag>
+ <item>The address of the fun's code when HiPE is enabled.</item>
+ <tag><em>Refc</em></tag>
+ <item>The number of references to the fun.</item>
+ </taglist>
+ </section>
+
+ <section>
+ <marker id="proc_data"></marker>
+ <title>Process Data</title>
+ <p>For each process there will be at least one <em>=proc_stack</em>
+ and one <em>=proc_heap</em> tag followed by the raw memory
+ information for the stack and heap of the process.</p>
+ <p>For each process there will also be a <em>=proc_messages</em>
+ tag if the process' message queue is non-empty and a
+ <em>=proc_dictionary</em> tag if the process' dictionary (the
+ <c><![CDATA[put/2]]></c> and <c><![CDATA[get/1]]></c> thing) is non-empty.</p>
+ <p>The raw memory information can be decoded by the Crashdump
+ Viewer tool. You will then be able to see the stack dump, the
+ message queue (if any) and the dictionary (if any).</p>
+ <p>The stack dump is a dump of the Erlang process stack. Most of
+ the live data (i.e., variables currently in use) are placed on
+ the stack; thus this can be quite interesting. One has to
+ "guess" what's what, but as the information is symbolic,
+ thorough reading of this information can be very useful. As an
+ example we can find the state variable of the Erlang primitive
+ loader on line <c><![CDATA[(5)]]></c> in the example below:</p>
+ <code type="none"><![CDATA[
+(1) 3cac44 Return addr 0x13BF58 (<terminate process normally>)
+(2) y(0) ["/view/siri_r10_dev/clearcase/otp/erts/lib/kernel/ebin","/view/siri_r10_dev/
+(3) clearcase/otp/erts/lib/stdlib/ebin"]
+(4) y(1) <0.1.0>
+(5) y(2) {state,[],none,#Fun<erl_prim_loader.6.7085890>,undefined,#Fun<erl_prim_loader.7.9000327>,#Fun<erl_prim_loader.8.116480692>,#Port<0.2>,infinity,#Fun<erl_prim_loader.9.10708760>}
+(6) y(3) infinity ]]></code>
+ <p>When interpreting the data for a process, it is helpful to know
+ that anonymous function objects (funs) are given a name
+ constructed from the name of the function in which they are
+ created, and a number (starting with 0) indicating the number of
+ that fun within that function.</p>
+ </section>
+
+ <section>
+ <marker id="atoms"></marker>
+ <title>Atoms</title>
+ <p>Now all the atoms in the system are written. This is only
+ interesting if one suspects that dynamic generation of atoms could
+ be a problem, otherwise this section can be ignored.</p>
+ <p>Note that the last created atom is printed first.</p>
+ </section>
+
+ <section>
+ <title>Disclaimer</title>
+ <p>The format of the crash dump evolves between releases of
+ OTP. Some information here may not apply to your
+ version. A description as this will never be complete; it is meant as
+ an explanation of the crash dump in general and as a help
+ when trying to find application errors, not as a complete
+ specification.</p>
+ </section>
+</chapter>
+