Update file performance advice

The parts relating to drivers/ports are now obsolete, and the
provided example was far noisier than it had to be; the only
relevant metric is the number of calls and it's up to the user to
decide how those will be reduced. One could argue for its complete
removal, but I'm inclined to leave it be.

1 files changed, 82 insertions, 138 deletions

diff --git a/lib/kernel/doc/src/file.xml b/lib/kernel/doc/src/file.xml
index 2ab35b9b05..3e3c5b3bf3 100644
--- a/lib/kernel/doc/src/file.xml
+++ b/lib/kernel/doc/src/file.xml
@@ -33,11 +33,14 @@
   <description>
     <p>This module provides an interface to the file system.</p>
 
-    <p>On operating systems with thread support,
-      file operations can be performed in threads of their own, allowing
-      other Erlang processes to continue executing in parallel with
-      the file operations. See command-line flag
-      <c>+A</c> in <seealso marker="erts:erl"><c>erl(1)</c></seealso>.</p>
+    <warning>
+       <p>File operations are only guaranteed to appear atomic when going
+          through the same file server. A NIF or other OS process may observe
+          intermediate steps on certain operations on some operating systems,
+          eg. renaming an existing file on Windows, or
+          <seealso marker="#write_file_info/2"><c>write_file_info/2</c>
+          </seealso> on any OS at the time of writing.</p>
+    </warning>
 
     <p>Regarding filename encoding, the Erlang VM can operate in
     two modes. The current mode can be queried using function
@@ -1438,8 +1441,12 @@ f.txt:  {person, "kalle", 25}.
 	    which is 1970-01-01 00:00 UTC.</p></item>
 	  </taglist>
 	<p>Default is <c>{time, local}</c>.</p>
-        <p>If the option <c>raw</c> is set, the file server is not called
-          and only information about local files is returned.</p>
+        <p>If the option <c>raw</c> is set, the file server is not called and
+           only information about local files is returned. Note that this will
+           break this module's atomicity guarantees as it can race with a
+           concurrent call to
+           <seealso marker="#write_file_info/2"><c>write_file_info/1,2</c>
+           </seealso></p>
 	  <note>
           <p>As file times are stored in POSIX time on most OS, it is faster to
 	  query file information with option <c>posix</c>.</p>
@@ -1687,8 +1694,12 @@ f.txt:  {person, "kalle", 25}.
 	except that if <c><anno>Name</anno></c> is a symbolic link, information
 	about the link is returned in the <c>file_info</c> record and
           the <c>type</c> field of the record is set to <c>symlink</c>.</p>
-      <p>If the option <c>raw</c> is set, the file server is not called
-        and only information about local files is returned.</p>
+        <p>If the option <c>raw</c> is set, the file server is not called and
+           only information about local files is returned. Note that this will
+           break this module's atomicity guarantees as it can race with a
+           concurrent call to
+           <seealso marker="#write_file_info/2"><c>write_file_info/1,2</c>
+           </seealso></p>
         <p>If <c><anno>Name</anno></c> is not a symbolic link, this function returns
           the same result as <c>read_file_info/1</c>.
           On platforms that do not support symbolic links, this function
@@ -2148,144 +2159,77 @@ f.txt:  {person, "kalle", 25}.
 
   <section>
     <title>Performance</title>
-    <p>Some operating system file operations, for example, a
-      <c>sync/1</c> or <c>close/1</c> on a huge file, can block their
-      calling thread for seconds. If this affects the emulator main
-      thread, the response time is no longer in the order of
-      milliseconds, depending on the definition of "soft" in soft
-      real-time system.</p>
-    <p>If the device driver thread pool is active, file operations are
-      done through those threads instead, so the emulator can go on
-      executing Erlang processes. Unfortunately, the time for serving a
-      file operation increases because of the extra scheduling required
-      from the operating system.</p>
-    <p>If the device driver thread pool is disabled or of size 0, large
-      file reads and writes are segmented into many smaller, which
-      enable the emulator to serve other processes during the file
-      operation. This has the same effect as when using the thread
-      pool, but with larger overhead. Other file operations, for
-      example, <c>sync/1</c> or <c>close/1</c> on a huge file, still are
-      a problem.</p>
-    <p>For increased performance, raw files are recommended. Raw files
-      use the file system of the host machine of the node.</p>
+    <p>For increased performance, raw files are recommended.</p>
+    <p>A normal file is really a process so it can be used as an I/O
+       device (see <seealso marker="stdlib:io"><c>io</c></seealso>).
+       Therefore, when data is written to a normal file, the sending of the
+       data to the file process, copies all data that are not binaries. Opening
+       the file in binary mode and writing binaries is therefore recommended.
+       If the file is opened on another node, or if the file server runs as
+       slave to the file server of another node, also binaries are copied.</p>
     <note>
-      <p>
-	For normal files (non-raw), the file server is used to find the files,
-	and if the node is running its file server as slave to the file server
-	of another node, and the other node runs on some other host machine,
-	they can have different file systems.
-        However, this is seldom a problem.</p>
+      <p>Raw files use the file system of the host machine of the node.
+         For normal files (non-raw), the file server is used to find the files,
+         and if the node is running its file server as slave to the file server
+         of another node, and the other node runs on some other host machine,
+         they can have different file systems.
+         However, this is seldom a problem.</p>
     </note>
-    <p>A normal file is really a process so it can be used as an I/O
-      device (see
-      <seealso marker="stdlib:io"><c>io</c></seealso>).
-      Therefore, when data is written to a
-      normal file, the sending of the data to the file process, copies 
-      all data that are not binaries. Opening the file in binary mode
-      and writing binaries is therefore recommended. If the file is
-      opened on another node, or if the file server runs as slave to
-      the file server of another node, also binaries are copied.</p>
-    <p>Caching data to reduce the number of file operations, or rather
-      the number of calls to the file driver, generally increases
-      performance. The following function writes 4 MBytes in 23
-      seconds when tested:</p>
+    <p><seealso marker="#open/2"><c>open/2</c></seealso> can be given the
+       options <c>delayed_write</c> and <c>read_ahead</c> to turn on caching,
+       which will reduce the number of operating system calls and greatly
+       improve performance for small reads and writes. However, the overhead
+       won't disappear completely and it's best to keep the number of file
+       operations to a minimum. As a contrived example, the following function
+       writes 4MB in 2.5 seconds when tested:</p>
+
     <code type="none"><![CDATA[
-create_file_slow(Name, N) when integer(N), N >= 0 ->
-    {ok, FD} = file:open(Name, [raw, write, delayed_write, binary]),
-    ok = create_file_slow(FD, 0, N),
-    ok = ?FILE_MODULE:close(FD),
-    ok.
-      
-create_file_slow(FD, M, M) ->
+create_file_slow(Name) ->
+    {ok, Fd} = file:open(Name, [raw, write, delayed_write, binary]),
+    create_file_slow_1(Fd, 4 bsl 20),
+    file:close(Fd).
+
+create_file_slow_1(_Fd, 0) ->
     ok;
-create_file_slow(FD, M, N) ->
-    ok = file:write(FD, <<M:32/unsigned>>),
-    create_file_slow(FD, M+1, N).]]></code>
+create_file_slow_1(Fd, M) ->
+    ok = file:write(Fd, <<0>>),
+    create_file_slow_1(Fd, M - 1).]]></code>
+
+    <p>The following functionally equivalent code writes 128 bytes per call
+       to <seealso marker="#write/2"><c>write/2</c></seealso> and so does the
+       same work in 0.08 seconds, which is roughly 30 times faster:</p>
 
-    <p>The following, functionally equivalent, function collects 1024
-      entries into a list of 128 32-byte binaries before each call to
-      <seealso marker="#write/2"><c>write/2</c></seealso> and so
-      does the same work in 0.52 seconds,
-      which is 44 times faster:</p>
     <code type="none"><![CDATA[
-create_file(Name, N) when integer(N), N >= 0 ->
-    {ok, FD} = file:open(Name, [raw, write, delayed_write, binary]),
-    ok = create_file(FD, 0, N),
-    ok = ?FILE_MODULE:close(FD),
+create_file(Name) ->
+    {ok, Fd} = file:open(Name, [raw, write, delayed_write, binary]),
+    create_file_1(Fd, 4 bsl 20),
+    file:close(Fd),
     ok.
-      
-create_file(FD, M, M) ->
+
+create_file_1(_Fd, 0) ->
     ok;
-create_file(FD, M, N) when M + 1024 =&lt; N ->
-    create_file(FD, M, M + 1024, []),
-    create_file(FD, M + 1024, N);
-create_file(FD, M, N) ->
-    create_file(FD, M, N, []).
-      
-create_file(FD, M, M, R) ->
-    ok = file:write(FD, R);
-create_file(FD, M, N0, R) when M + 8 =&lt; N0 ->
-    N1  = N0-1,  N2  = N0-2,  N3  = N0-3,  N4  = N0-4, 
-    N5  = N0-5,  N6  = N0-6,  N7  = N0-7,  N8  = N0-8, 
-    create_file(FD, M, N8, 
-                [<<N8:32/unsigned,  N7:32/unsigned, 
-                   N6:32/unsigned,  N5:32/unsigned, 
-                   N4:32/unsigned,  N3:32/unsigned, 
-                   N2:32/unsigned,  N1:32/unsigned>> | R]);
-create_file(FD, M, N0, R) ->
-    N1 = N0-1,
-    create_file(FD, M, N1, [<<N1:32/unsigned>> | R]).]]></code>
+create_file_1(Fd, M) when M >= 128 ->
+    ok = file:write(Fd, <<0:(128)/unit:8>>),
+    create_file_1(Fd, M - 128);
+create_file_1(Fd, M) ->
+    ok = file:write(Fd, <<0:(M)/unit:8>>),
+    create_file_1(Fd, M - 1).]]></code>
 
-    <note>
-      <p>Trust only your own benchmarks. If the list length in 
-        <c>create_file/2</c> above is increased, it runs slightly
-        faster, but consumes more memory and causes more memory
-        fragmentation. How much this affects your application is
-        something that this simple benchmark cannot predict.</p>
-      <p>If the size of each binary is increased to 64 bytes, it
-        also runs slightly faster, but the code is then twice as clumsy.
-        In the current implementation, binaries larger than 64 bytes are
-        stored in memory common to all processes and not copied when
-        sent between processes, while these smaller binaries are stored
-        on the process heap and copied when sent like any other term.</p>
-      <p>So, with a binary size of 68 bytes, <c>create_file/2</c> runs
-        30 percent slower than with 64 bytes, and causes much more
-        memory fragmentation. Notice that if the binaries were to be sent
-        between processes (for example, a non-raw file), the results
-        would probably be completely different.</p>
-    </note>
-    <p>A raw file is really a port. When writing data to a port, it is
-      efficient to write a list of binaries. It is not needed to
-      flatten a deep list before writing. On Unix hosts, scatter output,
-      which writes a set of buffers in one operation, is used when
-      possible. In this way <c>write(FD, [Bin1, Bin2 | Bin3])</c>
-      writes the contents of the binaries without copying the data
-      at all, except for perhaps deep down in the operating system
-      kernel.</p>
-    <p>For raw files, <c>pwrite/2</c> and <c>pread/2</c> are
-      efficiently implemented. The file driver is called only once for
-      the whole operation, and the list iteration is done in the file
-      driver.</p>
-    <p>The options <c>delayed_write</c> and <c>read_ahead</c> to 
-      <seealso marker="#open/2"><c>open/2</c></seealso>
-      make the file driver cache data to reduce
-      the number of operating system calls. The function
-      <c>create_file/2</c> in the recent example takes 60 seconds
-      without option <c>delayed_write</c>, which is 2.6
-      times slower.</p>
-    <p>As a bad example, <c>create_file_slow/2</c>
-      without options <c>raw</c>, <c>binary</c>, and <c>delayed_write</c>,
-      meaning it calls <c>open(Name, [write])</c>, needs
-      1 min 20 seconds for the job, which is 3.5 times slower than
-      the first example, and 150 times slower than the optimized
-      <c>create_file/2</c>.</p>
-      <warning>
-	<p>If an error occurs when accessing an open file with module
-      <seealso marker="stdlib:io"><c>io</c></seealso>,
-      the process handling the file exits. The dead
-      file process can hang if a process tries to access it later.
-      This will be fixed in a future release.</p>
-      </warning>
+    <p>When writing data it's generally more efficient to write a list of
+       binaries rather than a list of integers. It is not needed to
+       flatten a deep list before writing. On Unix hosts, scatter output,
+       which writes a set of buffers in one operation, is used when
+       possible. In this way <c>write(FD, [Bin1, Bin2 | Bin3])</c>
+       writes the contents of the binaries without copying the data
+       at all, except for perhaps deep down in the operating system
+       kernel.</p>
+    <warning>
+        <p>If an error occurs when accessing an open file with module
+           <seealso marker="stdlib:io"><c>io</c></seealso>, the process
+           handling the file exits. The dead file process can hang if a process
+           tries to access it later. This will be fixed in a future release.
+           </p>
+    </warning>
   </section>
 
   <section>