Merge branch 'hb/unicode/OTP-10302'

OTP-10742
OTP-10745
OTP-10749

* hb/unicode/OTP-10302:
  [stdlib] Remove documentation of ~tp
  Remove one use of iolist_size/1 in io_lib_pretty.erl
  Add a new function proc_lib:format/2 which takes encoding
  Export the type erl_scan:token()
  Make adjustments for Unicode
  [stdlib] Change default of erl_scan's unicode option
  Update preloaded
  Correct recently introduced Unicode related type errors
  [stdlib] Introduce new functions epp:read_encoding_from_binary/1,2
  Extend char() to Unicode characters
  [kernel] Correct bugs in the old shell (user.erl)
  [parsetools] Change the encoding of test suites to UTF-8
  [stdlib] Fix a contract bug
  [stdlib] Update the Unicode examples in STDLIB User's Guide
  [stdlib] Remove documentation of Unicode functions in io_lib

1 files changed, 51 insertions, 25 deletions

diff --git a/lib/stdlib/doc/src/io.xml b/lib/stdlib/doc/src/io.xml
index 22cd45a482..fa475804eb 100644
--- a/lib/stdlib/doc/src/io.xml
+++ b/lib/stdlib/doc/src/io.xml
@@ -4,7 +4,7 @@
 <erlref>
   <header>
     <copyright>
-      <year>1996</year><year>2012</year>
+      <year>1996</year><year>2013</year>
       <holder>Ericsson AB. All Rights Reserved.</holder>
     </copyright>
     <legalnotice>
@@ -306,6 +306,11 @@
           <item>
             <p>The parsing failed.</p>
           </item>
+	  <tag><c>{error, <anno>ErrorDescription</anno>}</c></tag>
+	  <item>
+	    <p>Other (rare) error condition, for instance <c>{error, estale}</c>
+	    if reading from an NFS file system.</p>
+	  </item>
         </taglist>
       </desc>
     </func>
@@ -333,6 +338,11 @@
           <item>
             <p>The parsing failed.</p>
           </item>
+	  <tag><c>{error, <anno>ErrorDescription</anno>}</c></tag>
+	  <item>
+	    <p>Other (rare) error condition, for instance <c>{error, estale}</c>
+	    if reading from an NFS file system.</p>
+	  </item>
         </taglist>
       </desc>
     </func>
@@ -440,9 +450,12 @@ ok</pre>
           </item>
           <tag><c>s</c></tag>
           <item>
-            <p>Prints the argument with the <c>string</c> syntax. The
+            <p>Prints the argument with the string syntax. The
               argument is, if no Unicode translation modifier is present, an 
-              iolist(), a binary, or an atom. If the Unicode translation modifier (<c>t</c>) is in effect, the argument is unicode:chardata(), meaning that binaries are in UTF-8. The characters
+              <c>iolist()</c>, a <c>binary()</c>, or an <c>atom()</c>.
+              If the Unicode translation modifier (<c>t</c>) is in effect,
+              the argument is <c>unicode:chardata()</c>, meaning that
+              binaries are in UTF-8. The characters
               are printed without quotes. The string is first truncated
 	      by the given precision and then padded and justified
 	      to the given field width. The default precision is the field width.</p>
@@ -480,9 +493,8 @@ ok
               <c>~w</c>, but breaks terms whose printed representation
               is longer than one line into many lines and indents each
               line sensibly. It also tries to detect lists of
-              printable characters and to output these as strings. The
-              Unicode translation modifier is used for determining
-              what characters are printable. For example:</p>
+              printable characters and to output these as strings.
+              For example:</p>
             <pre>
 5> <input>T = [{attributes,[[{id,age,1.50000},{mode,explicit},</input>
 <input>{typename,"INTEGER"}], [{id,cho},{mode,explicit},{typename,'Cho'}]]},</input>
@@ -521,19 +533,6 @@ Here T = [{attributes,[[{id,age,1.5},
           {tag,{'PRIVATE',3}},
           {mode,implicit}]
 ok</pre>
-            <p>Binaries that look like UTF-8 encoded strings will be
-              output with the string syntax if the Unicode translation
-              modifier is given:</p>
-            <pre>
-9> <input>io:fwrite("~p~n",[[1024]]).</input>
-[1024]
-10> <input>io:fwrite("~tp~n",[[1024]]).</input>
-"\x{400}"
-11> <input>io:fwrite("~tp~n", [&lt;&lt;128,128&gt;&gt;]).</input>
-&lt;&lt;128,128&gt;&gt;
-12> <input>io:fwrite("~tp~n", [&lt;&lt;208,128&gt;&gt;]).</input>
-&lt;&lt;"\x{400}"/utf8&gt;&gt;
-ok</pre>
           </item>
           <tag><c>W</c></tag>
           <item>
@@ -848,12 +847,21 @@ enter><input>:</input>   <input>alan</input>   <input>:</input>   <input>joe</in
           </item>
           <tag><c>{eof, EndLocation}</c></tag>
           <item>
-            <p>End of file was encountered.</p>
+            <p>End of file was encountered by the tokenizer.</p>
+          </item>
+          <tag><c>eof</c></tag>
+          <item>
+            <p>End of file was encountered by the I/O-server.</p>
           </item>
           <tag><c>{error, ErrorInfo, ErrorLocation}</c></tag>
           <item>
-            <p>An error occurred.</p>
+            <p>An error occurred while tokenizing.</p>
           </item>
+	  <tag><c>{error, <anno>ErrorDescription</anno>}</c></tag>
+	  <item>
+	    <p>Other (rare) error condition, for instance <c>{error, estale}</c>
+	    if reading from an NFS file system.</p>
+	  </item>
         </taglist>
         <p>Example:</p>
         <pre>
@@ -910,12 +918,21 @@ enter><input>1.0er.</input>
           </item>
           <tag><c>{eof, EndLocation}</c></tag>
           <item>
-            <p>End of file was encountered.</p>
+            <p>End of file was encountered by the tokenizer.</p>
+          </item>
+          <tag><c>eof</c></tag>
+          <item>
+            <p>End of file was encountered by the I/O-server.</p>
           </item>
           <tag><c>{error, ErrorInfo, ErrorLocation}</c></tag>
           <item>
-            <p>An error occurred.</p>
+            <p>An error occurred while tokenizing or parsing.</p>
           </item>
+	  <tag><c>{error, <anno>ErrorDescription</anno>}</c></tag>
+	  <item>
+	    <p>Other (rare) error condition, for instance <c>{error, estale}</c>
+	    if reading from an NFS file system.</p>
+	  </item>
         </taglist>
         <p>Example:</p>
         <pre>
@@ -952,12 +969,21 @@ enter><input>abc("hey".</input>
           </item>
           <tag><c>{eof, EndLocation}</c></tag>
           <item>
-            <p>End of file was encountered.</p>
+            <p>End of file was encountered by the tokenizer.</p>
+          </item>
+          <tag><c>eof</c></tag>
+          <item>
+            <p>End of file was encountered by the I/O-server.</p>
           </item>
           <tag><c>{error, ErrorInfo, ErrorLocation}</c></tag>
           <item>
-            <p>An error occurred.</p>
+            <p>An error occurred while tokenizing or parsing.</p>
           </item>
+	  <tag><c>{error, <anno>ErrorDescription</anno>}</c></tag>
+	  <item>
+	    <p>Other (rare) error condition, for instance <c>{error, estale}</c>
+	    if reading from an NFS file system.</p>
+	  </item>
         </taglist>
       </desc>
     </func>