erts: Improve crash dumps

This commit improves crash dumps in several ways:
  *  Suspends schedulers to get a current snapshot
  *  Dumps information about scheduler
  *  Dumps stack trace of current running process
     (including Garbing processes)

1 files changed, 84 insertions, 4 deletions

diff --git a/erts/doc/src/crash_dump.xml b/erts/doc/src/crash_dump.xml
index 2b5fc877c3..8291bf38b7 100644
--- a/erts/doc/src/crash_dump.xml
+++ b/erts/doc/src/crash_dump.xml
@@ -55,10 +55,12 @@
     emulator or the operating system can be reconfigured to avoid the
     crash, which is why interpreting the crash dump correctly is
     important.</p>
+  <p>On systems that support OS signals, it is also possible to stop
+    the runtime system and generate a crash dump by sending the SIGUSR1.</p>
   <p>The erlang crash dump is a readable text file, but it might not be
     very easy to read. Using the Crashdump Viewer tool in the
     <c><![CDATA[observer]]></c> application will simplify the task. This is an
-    HTML based tool for browsing Erlang crash dumps.</p>
+    wx-widget based tool for browsing Erlang crash dumps.</p>
 
   <section>
     <marker id="general_info"></marker>
@@ -66,8 +68,9 @@
     <p>The first part of the dump shows the creation time for the dump,
       a slogan indicating the reason for the dump, the system version,
       of the node from which the dump originates, the compile time of
-      the emulator running the originating node and the number of
-      atoms in the atom table.
+      the emulator running the originating node, the number of
+      atoms in the atom table and the runtime system thread that caused
+      the crash dump to happen.
       </p>
 
     <section>
@@ -170,6 +173,60 @@
   </section>
 
   <section>
+    <marker id="scheduler"></marker>
+    <title>Scheduler information</title>
+    <p>Under the tag <em>=scheduler</em> information about the current state
+    and statistics of the schedulers in the runtime system is displayed.
+    On OSs that do allow instant suspension of other threads, the data within
+    this section will reflect what the runtime system looks like at the moment
+    when the crash happens.</p>
+    <p>The following fields can exist for a process:</p>
+    <taglist>
+      <tag><em>=scheduler:id</em></tag>
+      <item>Header, states the scheduler identifier.</item>
+      <tag><em>Scheduler Sleep Info Flags</em></tag>
+      <item>If empty the scheduler was doing some work.
+      If not empty the scheduler is either in some state of sleep,
+      or suspended. This entry is only present in a SMP enabled emulator</item>
+      <tag><em>Scheduler Sleep Info Aux Work</em></tag>
+      <item>If not empty, a scheduler internal auxiliary work is scheduled
+      to be done.</item>
+      <tag><em>Current Port</em></tag>
+      <item>The port identifier of the port that is currently being
+      executed by the scheduler.</item>
+      <tag><em>Current Process</em></tag>
+      <item>The process identifier of the process that is currently being
+      executed by the scheduler. If there is such a process this entry is
+      followed by the <em>State</em>,<em>Internal State</em>,
+      <em>Program Counter</em>, <em>CP</em> of that same process. See
+      <seealso marker="#processes">Process Information</seealso> for a
+      description what the different entries mean. Keep in mind that
+      this is a snapshot of what the entries are exactly when the crash
+      dump is starting to be generated. Therefore they will most likely
+      be different (and more telling) then the entries for the same
+      processes found in the <em>=proc</em> section. If there is no currently
+      running process, only the <em>Current Process</em> entry will be printed.
+      </item>
+      <tag><em>Current Process Limited Stack Trace</em></tag>
+      <item>This entry only shows up if there is a current process. It is very
+      similar to <seealso marker="#proc_data"><em>=proc_stack</em></seealso>,
+      except that only the function frames are printed (i.e. the stack variables
+      are omited). It is also limited to only print the top and bottom part
+      of the stack. If the stack is small (less that 512 slots) then the
+      entire stack will be printed. If not, an entry stating
+      <code>skipping ## slots</code> will be printed where ## is
+      replaced by the number of slots that has been skipped.</item>
+      <tag><em>Run Queue</em></tag>
+      <item>Displays statistics about how many processes and ports
+      of different priorities are scheduled on this scheduler.</item>
+      <tag><em>** crashed **</em></tag>
+      <item>This entry is normally not printed. It signifies that getting
+      the rest of the information about this scheduler failed for some reason.
+      </item>
+    </taglist>
+  </section>
+
+  <section>
     <marker id="memory"></marker>
     <title>Memory information</title>
     <p>Under the tag <em>=memory</em> you will find information similar
@@ -314,6 +371,9 @@
       <item>The number of live argument registers. The argument registers,
        if any are live, will follow. These may contain the arguments
        of the function if they are not yet moved to the stack.</item>
+       <item><em>Internal State</em></item>
+       <item>A more detailed internal represantation of the state of
+       this process.</item>
     </taglist>
     <p>See also the section about <seealso marker="#proc_data">process data</seealso>.</p>
   </section>
@@ -339,18 +399,38 @@
       <tag><em>Name</em></tag>
       <item>The name of the table, regardless of whether it is a
       <c><![CDATA[named_table]]></c> or not.</item>
-      <tag><em>Buckets</em></tag>
+      <tag><em>Hash table, Buckets</em></tag>
       <item>This occurs if the table is a hash table, i.e. if it is not an
       <c><![CDATA[ordered_set]]></c>.</item>
+      <tag><em>Hash table, Chain Length</em></tag>
+      <item>Only applicable for hash tables. Contains statistics about the
+      hash table, such as the max, min and avg chain length. Having a max much
+      larger than the avg, and a std dev much larger that
+      the expected std dev is a sign that the hashing of the terms is
+      behaving badly for some reason.</item>
       <tag><em>Ordered set (AVL tree), Elements</em></tag>
       <item>This occurs only if the table is an <c><![CDATA[ordered_set]]></c>. (The
        number of elements is the same as the number of objects in the
        table.)</item>
+      <tag><em>Fixed</em></tag>
+      <item>If the table is fixed using ets:safe_fixtable or some internal
+      mechanism.</item>
       <tag><em>Objects</em></tag>
       <item>The number of objects in the table</item>
       <tag><em>Words</em></tag>
       <item>The number of words (usually 4 bytes/word) allocated to data
        in the table.</item>
+      <tag><em>Type</em></tag>
+      <item>The type of the table, i.e. <c>set</c>, <c>bag</c>,
+      <c>dublicate_bag</c> or <c>ordered_set</c>.</item>
+      <tag><em>Compressed</em></tag>
+      <item>If this table was compressed.</item>
+      <tag><em>Protection</em></tag>
+      <item>The protection of this table.</item>
+      <tag><em>Write Concurrency</em></tag>
+      <item>If write_concurrency was enabled for this table.</item>
+      <tag><em>Read Concurrency</em></tag>
+      <item>If read_concurrency was enabled for this table.</item>
     </taglist>
   </section>