Merge branch 'vinoski/ds'

* vinoski/ds:
  initial support for dirty schedulers and dirty NIFs

3 files changed, 262 insertions, 8 deletions

diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml
index e737727941..27a23174d5 100644
--- a/erts/doc/src/erl.xml
+++ b/erts/doc/src/erl.xml
@@ -792,6 +792,54 @@
           SMP support enabled (see the <seealso marker="#smp">-smp</seealso>
           flag).</p>
       </item>
+      <tag><marker id="+SDcpu"><c><![CDATA[+SDcpu DirtyCPUSchedulers:DirtyCPUSchedulersOnline]]></c></marker></tag>
+      <item>
+          <p>Sets the number of dirty CPU scheduler threads to create and dirty
+            CPU scheduler threads to set online when threading support has been
+            enabled. The maximum for both values is 1024, and each value is further
+            limited by the settings for normal schedulers: the number of dirty CPU
+            scheduler threads created cannot exceed the number of normal scheduler
+            threads created, and the number of dirty CPU scheduler threads online
+            cannot exceed the number of normal scheduler threads online (see the
+            <seealso marker="#+S">+S</seealso> and <seealso marker="#+SP">+SP</seealso>
+            flags for more details). By default, the number of dirty CPU scheduler
+            threads created equals the number of normal scheduler threads created,
+            and the number of dirty CPU scheduler threads online equals the number
+            of normal scheduler threads online. <c>DirtyCPUSchedulers</c> may be
+            omitted if <c>:DirtyCPUSchedulersOnline</c> is not and vice versa. The
+            number of dirty CPU schedulers online can be changed at run time via
+            <seealso marker="erlang#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
+          </p>
+          <p>This option is ignored if the emulator doesn't have threading support
+            enabled. Currently, <em>this option is experimental</em> and is supported only
+            if the emulator was configured and built with support for dirty schedulers
+            enabled (it's disabled by default).
+          </p>
+      </item>
+      <tag><marker id="+SDPcpu"><c><![CDATA[+SDPcpu DirtyCPUSchedulersPercentage:DirtyCPUSchedulersOnlinePercentage]]></c></marker></tag>
+      <item>
+          <p>Similar to <seealso marker="#+SDcpu">+SDcpu</seealso> but uses percentages to set the
+            number of dirty CPU scheduler threads to create and number of dirty CPU scheduler threads
+            to set online when threading support has been enabled. Specified values must be greater
+            than 0. For example, <c>+SDPcpu 50:25</c> sets the number of dirty CPU scheduler threads
+            to 50% of the logical processors configured and the number of dirty CPU scheduler threads
+            online to 25% of the logical processors available. <c>DirtyCPUSchedulersPercentage</c> may
+            be omitted if <c>:DirtyCPUSchedulersOnlinePercentage</c> is not and vice versa. The
+            number of dirty CPU schedulers online can be changed at run time via
+            <seealso marker="erlang#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
+          </p>
+          <p>This option interacts with <seealso marker="#+SDcpu">+SDcpu</seealso> settings.
+            For example, on a system with 8 logical cores configured and 8 logical cores available,
+            the combination of the options <c>+SDcpu 4:4 +SDPcpu 50:25</c> (in either order) results
+            in 2 dirty CPU scheduler threads (50% of 4) and 1 dirty CPU scheduler thread online (25% of 4).
+          </p>
+          <p>This option is ignored if the emulator doesn't have threading support
+            enabled. Currently, <em>this option is experimental</em> and is supported only
+            if the emulator was configured and built with support for dirty schedulers
+            enabled (it's disabled by default).
+          </p>
+      </item>
+      <tag><marker id="+SDio"><c><![CDATA[+SDio IOSchedulers]]></c></marker></tag>
       <tag><c><![CDATA[+sFlag Value]]></c></tag>
       <item>
         <p>Scheduling specific flags.</p>
diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml
index 7ac8181d47..8b19725c02 100644
--- a/erts/doc/src/erl_nif.xml
+++ b/erts/doc/src/erl_nif.xml
@@ -181,7 +181,11 @@ ok
      to dispatch the work to another thread, return
      from the native function, and wait for the result. The thread can send
      the result back to the calling thread using message passing. Information
-     about thread primitives can be found below.</p>
+     about thread primitives can be found below. If you have built your system
+     with <em>the currently experimental</em> support for dirty schedulers,
+     you may want to try out this functionality by dispatching the work to a
+     <seealso marker="#dirty_nifs">dirty NIF</seealso>,
+     which does not have the same duration restriction as a normal NIF.</p>
   </description>
   <section>
   <title>FUNCTIONALITY</title>
@@ -312,6 +316,38 @@ ok
       <p>The library initialization callbacks <c>load</c>, <c>reload</c> and
       <c>upgrade</c> are all thread-safe even for shared state data.</p>
       </item>
+      <tag>Dirty NIFs</tag>
+      <item><p><marker id="dirty_nifs"/><em>Note that the dirty NIF functionality
+      is experimental</em> and that you have to enable support for dirty
+      schedulers when building OTP in order to try the functionality out. Native functions
+      <seealso marker="#lengthy_work">
+      must normally run quickly</seealso>, as explained earlier in this document. They
+      generally should execute for no more than a millisecond. But not all native functions
+      can execute so quickly; for example, functions that encrypt large blocks of data or
+      perform lengthy file system operations can often run for tens of seconds or more.</p>
+      <p>A NIF that cannot execute in a millisecond or less is called a "dirty NIF" since
+      it performs work that the Erlang runtime cannot handle cleanly. Applications
+      that make use of such functions must indicate to the runtime that the functions are
+      dirty so they can be handled specially. To schedule a dirty NIF for execution, the
+      application calls <seealso marker="#enif_schedule_dirty_nif">enif_schedule_dirty_nif</seealso>,
+      passing to it a pointer to the dirty NIF to be executed and indicating with a flag
+      argument whether it expects the operation to be CPU-bound or I/O-bound.</p>
+      <p>All dirty NIFs must ultimately invoke the <seealso marker="#enif_schedule_dirty_nif_finalizer">
+      enif_schedule_dirty_nif_finalizer</seealso> as their final action, passing to it the
+      result they wish to return to the original caller. A finalizer function can either
+      receive the result and return it directly, or it can return a different value instead.
+      For convenience, the NIF API provides the <seealso marker="#enif_dirty_nif_finalizer">
+      enif_dirty_nif_finalizer</seealso> function that applications can use as a finalizer;
+      it simply returns its result argument.</p>
+      <note><p>Dirty NIF support is available only when the emulator is configured with dirty
+      schedulers enabled. This feature is currently disabled by default. To determine whether
+      the dirty NIF API is available, native code can check to see if the C preprocessor macro
+      <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined. Also, if the Erlang runtime was built
+      without threading support, dirty schedulers are disabled. To check at runtime for the presence
+      of dirty scheduler threads, code can call the <seealso marker="#enif_have_dirty_schedulers"><c>
+      enif_have_dirty_schedulers()</c></seealso> API function, which returns true if dirty
+      scheduler threads are present, false otherwise.</p></note>
+      </item>
     </taglist>
   </section>
   <section>
@@ -610,6 +646,18 @@ typedef enum {
     See also the <seealso marker="#WARNING">warning</seealso> text at the beginning of this document.</p>
     </desc>
     </func>
+    <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result)</nametext></name>
+      <fsummary>Simple dirty NIF result finalizer</fsummary>
+      <desc>
+          <p>A convenience function that a dirty NIF can use as a finalizer that simply
+          return its <c>result</c> argument as its return value. This function is provided
+          for dirty NIFs with results that should be returned directly to the original caller.</p>
+          <note><p>This function is available only when the emulator is configured with dirty
+          schedulers enabled. This feature is currently disabled by default. To determine whether
+          the dirty NIF API is available, native code can check to see if the C preprocessor macro
+          <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
+      </desc>
+    </func>
     <func><name><ret>int</ret><nametext>enif_equal_tids(ErlNifTid tid1, ErlNifTid tid2)</nametext></name>
     <fsummary></fsummary>
     <desc><p>Same as <seealso marker="erl_driver#erl_drv_equal_tids">erl_drv_equal_tids</seealso>.
@@ -730,6 +778,22 @@ typedef enum {
       and return true, or return false if <c>term</c> is not an unsigned integer or is
       outside the bounds of type <c>unsigned long</c>.</p></desc>
     </func>
+    <func><name><ret>int</ret><nametext>enif_have_dirty_schedulers()</nametext></name>
+      <fsummary>Runtime check for the presence of dirty scheduler threads</fsummary>
+      <desc>
+          <p>Check at runtime for the presence of dirty scheduler threads. If the emulator is
+          built with threading support, dirty scheduler threads are available and
+          <c>enif_have_dirty_schedulers()</c> returns true. If the emulator was built without
+          threading support, <c>enif_have_dirty_schedulers()</c> returns false.</p>
+          <p>If dirty scheduler threads are not available in the emulator, calls to
+          <c>enif_schedule_dirty_nif</c> and <c>enif_schedule_dirty_nif_finalizer</c> result in
+          the NIF and finalizer functions being called directly within the calling thread.</p>
+          <note><p>This function is available only when the emulator is configured with dirty
+          schedulers enabled. This feature is currently disabled by default. To determine whether
+          the dirty NIF API is available, native code can check to see if the C preprocessor macro
+          <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
+      </desc>
+    </func>
     <func><name><ret>int</ret><nametext>enif_inspect_binary(ErlNifEnv* env, ERL_NIF_TERM bin_term, ErlNifBinary* bin)</nametext></name>
       <fsummary>Inspect the content of a binary</fsummary>
       <desc><p>Initialize the structure pointed to by <c>bin</c> with 
@@ -777,6 +841,20 @@ typedef enum {
       Erlang operators <c>=:=</c> and
       <c>=/=</c>.</p></desc> 
     </func>
+    <func><name><ret>int</ret><nametext>enif_is_on_dirty_scheduler(ErlNifEnv* env)</nametext></name>
+      <fsummary>Check to see if executing on a dirty scheduler thread</fsummary>
+      <desc>
+          <p>Check to see if the current NIF is executing on a dirty scheduler thread. If the
+          emulator is built with threading support, calling <c>enif_is_on_dirty_scheduler</c>
+          from within a dirty NIF returns true. It returns false when the calling NIF is a regular
+          NIF or a NIF finalizer, both of which run on normal scheduler threads, or when the emulator
+          is built without threading support.</p>
+          <note><p>This function is available only when the emulator is configured with dirty
+          schedulers enabled. This feature is currently disabled by default. To determine whether
+          the dirty NIF API is available, native code can check to see if the C preprocessor macro
+          <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
+      </desc>
+    </func>
     <func><name><ret>int</ret><nametext>enif_is_pid(ErlNifEnv* env, ERL_NIF_TERM term)</nametext></name>
       <fsummary>Determine if a term is a pid</fsummary>
       <desc><p>Return true if <c>term</c> is a pid.</p></desc>
@@ -1141,6 +1219,48 @@ typedef enum {
     <desc><p>Same as <seealso marker="erl_driver#erl_drv_rwlock_tryrwlock">erl_drv_rwlock_tryrwlock</seealso>.
           </p></desc>
     </func>
+    <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_schedule_dirty_nif(ErlNifEnv* env, int flags, ERL_NIF_TERM (*fp)(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]), int argc, const ERL_NIF_TERM argv[])</nametext></name>
+      <fsummary>Schedule a dirty NIF for execution</fsummary>
+      <desc>
+          <p>Schedule dirty NIF <c>fp</c> to execute a long-running operation. The <c>flags</c>
+          argument must be set to either <c>ERL_NIF_DIRTY_JOB_CPU_BOUND</c> if the job is expected to
+          be primarily CPU-bound, or <c>ERL_NIF_DIRTY_JOB_IO_BOUND</c> for jobs that will be
+          I/O-bound. The <c>argc</c> and <c>argv</c> arguments can either be the originals passed
+          into the calling NIF, or they can be values created by the calling NIF. The calling
+          NIF must use the return value of <c>enif_schedule_dirty_nif</c> as its own return value.</p>
+          <p>Be aware that <c>enif_schedule_dirty_nif</c>, as its name implies, only schedules the
+          dirty NIF for future execution. The calling NIF does not block waiting for the dirty NIF to
+          execute and return, which means that the calling NIF can't expect to receive the dirty NIF
+          return value and use it for further operations.</p>
+          <p>A dirty NIF may not invoke the <seealso marker="#enif_make_badarg">enif_make_badarg</seealso>
+          to raise an exception. If it wishes to return an exception, the dirty NIF should pass a
+          regular result indicating the exception details to its finalizer, and allow the finalizer
+          to raise the exception on its behalf.</p>
+          <note><p>This function is available only when the emulator is configured with dirty schedulers
+          enabled. This feature is currently disabled by default. To determine whether the dirty NIF API
+          is available, native code can check to see if the C preprocessor macro
+          <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
+      </desc>
+    </func>
+    <func><name><ret>ERL_NIF_TERM</ret><nametext>enif_schedule_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result, ERL_NIF_TERM (*fp)(ErlNifEnv* env, ERL_NIF_TERM result))</nametext></name>
+      <fsummary>Schedule a dirty NIF finalizer</fsummary>
+      <desc>
+          <p>When a dirty NIF finishes executing, it must schedule a finalizer function to return
+          its result to the original NIF caller. The dirty NIF passes <c>result</c> as the value it
+          wants the finalizer to use as the return value. The <c>fp</c> argument is a pointer to the
+          finalizer function. The NIF API provides the <seealso marker="#enif_dirty_nif_finalizer">
+          enif_dirty_nif_finalizer</seealso> function that can be used as a finalizer that simply
+          returns its <c>result</c> argument. You are also free to write your own custom finalizer
+          that uses <c>result</c> to derive a different return value, or ignores <c>result</c>
+          entirely and returns a completely different value.</p>
+          <p>Without exception, all dirty NIFs must invoke <c>enif_schedule_dirty_nif_finalizer</c>
+          to complete their execution.</p>
+          <note><p>This function is available only when the emulator is configured with dirty
+          schedulers enabled. This feature is currently disabled by default. To determine whether
+          the dirty NIF API is available, native code can check to see if the C preprocessor macro
+          <c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
+      </desc>
+    </func>
     <func><name><ret>ErlNifPid *</ret><nametext>enif_self(ErlNifEnv* caller_env, ErlNifPid* pid)</nametext></name>
       <fsummary>Get the pid of the calling process.</fsummary>
       <desc><p>Initialize the pid variable <c>*pid</c> to represent the
diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml
index ea753cfaaf..4cf5631727 100644
--- a/erts/doc/src/erlang.xml
+++ b/erts/doc/src/erlang.xml
@@ -5194,6 +5194,27 @@ ok
     </func>
     <func>
       <name name="system_flag" arity="2" clause_i="3"/>
+      <fsummary>Set system flag dirty CPU schedulers online</fsummary>
+      <desc>
+        <p><marker id="system_flag_dirty_cpu_schedulers_online"></marker>
+        Sets the amount of dirty CPU schedulers online. Valid range is
+        <![CDATA[1 <= DirtyCPUSchedulersOnline <= N]]> where <c>N</c> is the
+        lesser of the return values of <c>erlang:system_info(dirty_cpu_schedulers)</c> and
+        <c>erlang:system_info(schedulers_online)</c>.
+        </p>
+        <p>Returns the old value of the flag.</p>
+	<p><em>Note that the dirty schedulers functionality is experimental</em>, and
+	that you have to enable support for dirty schedulers when building OTP in
+	order to try the functionality out.</p>
+        <p>For more information see
+        <seealso marker="#system_info_dirty_cpu_schedulers">erlang:system_info(dirty_cpu_schedulers)</seealso>
+        and
+        <seealso marker="#system_info_dirty_cpu_schedulers_online">erlang:system_info(dirty_cpu_schedulers_online)</seealso>.
+        </p>
+      </desc>
+    </func>
+    <func>
+      <name name="system_flag" arity="2" clause_i="4"/>
       <fsummary>Set system flag fullsweep_after</fsummary>
       <desc>
         <p><c><anno>Number</anno></c> is a non-negative integer which indicates
@@ -5211,7 +5232,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="4"/>
+      <name name="system_flag" arity="2" clause_i="5"/>
       <fsummary>Set system flag min_heap_size</fsummary>
       <desc>
         <p>Sets the default minimum heap size for processes. The
@@ -5226,7 +5247,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="5"/>
+      <name name="system_flag" arity="2" clause_i="6"/>
       <fsummary>Set system flag min_bin_vheap_size</fsummary>
       <desc>
         <p>Sets the default minimum binary virtual heap size for processes. The
@@ -5241,7 +5262,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="6"/>
+      <name name="system_flag" arity="2" clause_i="7"/>
       <fsummary>Set system flag multi_scheduling</fsummary>
       <desc>
         <p><marker id="system_flag_multi_scheduling"></marker>
@@ -5279,7 +5300,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="7"/>
+      <name name="system_flag" arity="2" clause_i="8"/>
       <type name="scheduler_bind_type"/>
       <fsummary>Set system flag scheduler_bind_type</fsummary>
       <desc>
@@ -5399,7 +5420,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="8"/>
+      <name name="system_flag" arity="2" clause_i="9"/>
       <fsummary>Set system flag scheduler_wall_time</fsummary>
       <desc><p><marker id="system_flag_scheduler_wall_time"></marker>
       Turns on/off scheduler wall time measurements. </p>
@@ -5409,7 +5430,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="9"/>
+      <name name="system_flag" arity="2" clause_i="10"/>
       <fsummary>Set system flag schedulers_online</fsummary>
       <desc>
         <p><marker id="system_flag_schedulers_online"></marker>
@@ -5425,7 +5446,7 @@ ok
       </desc>
     </func>
     <func>
-      <name name="system_flag" arity="2" clause_i="10"/>
+      <name name="system_flag" arity="2" clause_i="11"/>
       <fsummary>Set system flag trace_control_word</fsummary>
       <desc>
         <p>Sets the value of the node's trace control word to
@@ -5785,6 +5806,71 @@ ok
 	       compiled; otherwise, <c>false</c>.
 	    </p>
           </item>
+          <tag><marker id="system_info_dirty_cpu_schedulers"><c>dirty_cpu_schedulers</c></marker></tag>
+          <item>
+            <p>Returns the number of dirty CPU scheduler threads used by
+              the emulator. Dirty CPU schedulers execute CPU-bound
+              native functions such as NIFs, linked-in driver code, and BIFs
+              that cannot be managed cleanly by the emulator's normal schedulers.
+            </p>
+            <p>The number of dirty CPU scheduler threads is determined at emulator
+              boot time and cannot be changed after that. The number of dirty CPU
+              scheduler threads online can however be changed at any time. The number of
+              dirty CPU schedulers can be set on startup by passing
+              the <seealso marker="erts:erl#+SDcpu">+SDcpu</seealso> command line flag, see
+              <seealso marker="erts:erl#+SDcpu">erl(1)</seealso>.
+            </p>
+	    <p><em>Note that the dirty schedulers functionality is experimental</em>, and
+	    that you have to enable support for dirty schedulers when building OTP in
+	    order to try the functionality out.</p>
+            <p>See also <seealso marker="#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>,
+              <seealso marker="#system_info_dirty_cpu_schedulers_online">erlang:system_info(dirty_cpu_schedulers_online)</seealso>,
+              <seealso marker="#system_info_dirty_io_schedulers">erlang:system_info(dirty_io_schedulers)</seealso>,
+              <seealso marker="#system_info_schedulers">erlang:system_info(schedulers)</seealso>,
+	      <seealso marker="#system_info_schedulers_online">erlang:system_info(schedulers_online)</seealso>, and
+              <seealso marker="#system_flag_schedulers_online">erlang:system_flag(schedulers_online, SchedulersOnline)</seealso>.</p>
+          </item>
+          <tag><marker id="system_info_dirty_cpu_schedulers_online"><c>dirty_cpu_schedulers_online</c></marker></tag>
+          <item>
+            <p>Returns the number of dirty CPU schedulers online. The return value
+              satisfies the following relationship:
+              <c><![CDATA[1 <= DirtyCPUSchedulersOnline <= N]]></c>, where <c>N</c> is
+              the lesser of the return values of <c>erlang:system_info(dirty_cpu_schedulers)</c> and
+              <c>erlang:system_info(schedulers_online)</c>.
+            </p>
+            <p>The number of dirty CPU schedulers online can be set on startup by passing
+              the <seealso marker="erts:erl#+SDcpu">+SDcpu</seealso> command line flag, see
+              <seealso marker="erts:erl#+SDcpu">erl(1)</seealso>.
+            </p>
+	    <p><em>Note that the dirty schedulers functionality is experimental</em>, and
+	    that you have to enable support for dirty schedulers when building OTP in
+	    order to try the functionality out.</p>
+            <p>For more information, see
+              <seealso marker="#system_info_dirty_cpu_schedulers">erlang:system_info(dirty_cpu_schedulers)</seealso>,
+              <seealso marker="#system_info_dirty_io_schedulers">erlang:system_info(dirty_io_schedulers)</seealso>,
+              <seealso marker="#system_info_schedulers_online">erlang:system_info(schedulers_online)</seealso>, and
+              <seealso marker="#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
+            </p>
+          </item>
+          <tag><marker id="system_info_dirty_io_schedulers"><c>dirty_io_schedulers</c></marker></tag>
+          <item>
+            <p>Returns the number of dirty I/O schedulers as an integer. Dirty I/O schedulers
+              execute I/O-bound native functions such as NIFs and linked-in driver code that
+              cannot be managed cleanly by the emulator's normal schedulers.
+            </p>
+            <p>This value can be set on startup by passing
+              the <seealso marker="erts:erl#+SDio">+SDio</seealso> command line flag, see
+              <seealso marker="erts:erl#+SDio">erl(1)</seealso>.
+            </p>
+	    <p><em>Note that the dirty schedulers functionality is experimental</em>, and
+	    that you have to enable support for dirty schedulers when building OTP in
+	    order to try the functionality out.</p>
+            <p>For more information, see
+              <seealso marker="#system_info_dirty_cpu_schedulers">erlang:system_info(dirty_cpu_schedulers)</seealso>,
+              <seealso marker="#system_info_dirty_cpu_schedulers_online">erlang:system_info(dirty_cpu_schedulers_online)</seealso>, and
+              <seealso marker="#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
+            </p>
+          </item>
           <tag><c>dist</c></tag>
           <item>
             <p>Returns a binary containing a string of distribution