[stdlib, kernel] Introduce Unicode support for Erlang source files

Expect modifications, additions and corrections.

There is a kludge in file_io_server and
erl_scan:continuation_location() that's not so pleasing.

1 files changed, 49 insertions, 14 deletions

diff --git a/lib/stdlib/doc/src/io.xml b/lib/stdlib/doc/src/io.xml
index e6d262466c..ed425ce723 100644
--- a/lib/stdlib/doc/src/io.xml
+++ b/lib/stdlib/doc/src/io.xml
@@ -4,7 +4,7 @@
 <erlref>
   <header>
     <copyright>
-      <year>1996</year><year>2011</year>
+      <year>1996</year><year>2012</year>
       <holder>Ericsson AB. All Rights Reserved.</holder>
     </copyright>
     <legalnotice>
@@ -312,11 +312,14 @@
     </func>
     <func>
       <name name="read" arity="3"/>
+      <name name="read" arity="4"/>
       <fsummary>Read a term</fsummary>
       <desc>
         <p>Reads a term <c><anno>Term</anno></c> from <c><anno>IoDevice</anno></c>, prompting it
           with <c><anno>Prompt</anno></c>. Reading starts at line number
-          <c><anno>StartLine</anno></c>. It returns:</p>
+          <c><anno>StartLine</anno></c>. The argument
+          <c><anno>Options</anno></c> is passed on as the <c>Options</c>
+          argument of the <c>erl_scan:tokens/4</c> function. It returns:</p>
         <taglist>
           <tag><c>{ok, Term, <anno>EndLine</anno>}</c></tag>
           <item>
@@ -476,8 +479,10 @@ ok
             <p>Writes the data with standard syntax in the same way as
               <c>~w</c>, but breaks terms whose printed representation
               is longer than one line into many lines and indents each
-              line sensibly. It also tries to detect lists of printable
-              characters and to output these as strings. For example:</p>
+              line sensibly. It also tries to detect lists of
+              printable characters and to output these as strings. The
+              Unicode translation modifier is used for determining
+              what characters are printable. For example:</p>
             <pre>
 5> <input>T = [{attributes,[[{id,age,1.50000},{mode,explicit},</input>
 <input>{typename,"INTEGER"}], [{id,cho},{mode,explicit},{typename,'Cho'}]]},</input>
@@ -516,6 +521,19 @@ Here T = [{attributes,[[{id,age,1.5},
           {tag,{'PRIVATE',3}},
           {mode,implicit}]
 ok</pre>
+            <p>Binaries that look like UTF-8 encoded strings will be
+              output with the string syntax if the Unicode translation
+              modifier is given:</p>
+            <pre>
+9> <input>io:fwrite("~p~n",[[1024]]).</input>
+[1024]
+10> <input>io:fwrite("~tp~n",[[1024]]).</input>
+"\x{400}"
+11> <input>io:fwrite("~tp~n", [&lt;&lt;128,128&gt;&gt;]).</input>
+&lt;&lt;128,128&gt;&gt;
+12> <input>io:fwrite("~tp~n", [&lt;&lt;208,128&gt;&gt;]).</input>
+&lt;&lt;"\x{400}"/utf8&gt;&gt;
+ok</pre>
           </item>
           <tag><c>W</c></tag>
           <item>
@@ -805,11 +823,15 @@ enter><input>:</input>   <input>alan</input>   <input>:</input>   <input>joe</in
       <name name="scan_erl_exprs" arity="1"/>
       <name name="scan_erl_exprs" arity="2"/>
       <name name="scan_erl_exprs" arity="3"/>
+      <name name="scan_erl_exprs" arity="4"/>
       <fsummary>Read and tokenize Erlang expressions</fsummary>
       <desc>
         <p>Reads data from the standard input (<c>IoDevice</c>),
           prompting it with <c>Prompt</c>. Reading starts at line number
-          <c>StartLine</c> (1). The data is tokenized as if it were a
+          <c>StartLine</c> (1). The argument <c><anno>Options</anno></c>
+          is passed on as the <c>Options</c> argument of the
+          <c>erl_scan:tokens/4</c> function. The data is tokenized as if
+          it were a
           sequence of Erlang expressions until a final <c>'.'</c> is
           reached. This token is also returned. It returns:</p>
         <taglist>
@@ -840,11 +862,15 @@ enter><input>1.0er.</input>
       <name name="scan_erl_form" arity="1"/>
       <name name="scan_erl_form" arity="2"/>
       <name name="scan_erl_form" arity="3"/>
+      <name name="scan_erl_form" arity="4"/>
       <fsummary>Read and tokenize an Erlang form</fsummary>
       <desc>
         <p>Reads data from the standard input (<c><anno>IoDevice</anno></c>),
-          prompting it with <c><anno>Prompt</anno></c>. Starts reading at line number
-          <c><anno>StartLine</anno></c> (1). The data is tokenized as if it were an
+          prompting it with <c><anno>Prompt</anno></c>. Starts reading
+          at line number <c><anno>StartLine</anno></c> (1). The
+          argument <c><anno>Options</anno></c> is passed on as the
+          <c>Options</c> argument of the <c>erl_scan:tokens/4</c>
+          function. The data is tokenized as if it were an
           Erlang form - one of the valid Erlang expressions in an
           Erlang source file - until a final <c>'.'</c> is reached.
           This last token is also returned. The return values are the
@@ -855,14 +881,19 @@ enter><input>1.0er.</input>
       <name name="parse_erl_exprs" arity="1"/>
       <name name="parse_erl_exprs" arity="2"/>
       <name name="parse_erl_exprs" arity="3"/>
+      <name name="parse_erl_exprs" arity="4"/>
       <type name="parse_ret"/>
       <fsummary>Read, tokenize and parse Erlang expressions</fsummary>
       <desc>
-        <p>Reads data from the standard input (<c><anno>IoDevice</anno></c>),
-          prompting it with <c><anno>Prompt</anno></c>. Starts reading at line number
-          <c><anno>StartLine</anno></c> (1). The data is tokenized and parsed as if
-          it were a sequence of Erlang expressions until a final '.' is
-          reached. It returns:</p>
+        <p>Reads data from the standard input
+          (<c><anno>IoDevice</anno></c>), prompting it with
+          <c><anno>Prompt</anno></c>. Starts reading at line number
+          <c><anno>StartLine</anno></c> (1). The argument
+          <c><anno>Options</anno></c> is passed on as the
+          <c>Options</c> argument of the <c>erl_scan:tokens/4</c>
+          function. The data is tokenized and parsed as if it were a
+          sequence of Erlang expressions until a final '.' is reached.
+          It returns:</p>
         <taglist>
           <tag><c>{ok, ExprList, EndLine}</c></tag>
           <item>
@@ -891,12 +922,16 @@ enter><input>abc("hey".</input>
       <name name="parse_erl_form" arity="1"/>
       <name name="parse_erl_form" arity="2"/>
       <name name="parse_erl_form" arity="3"/>
+      <name name="parse_erl_form" arity="4"/>
       <type name="parse_form_ret"/>
       <fsummary>Read, tokenize and parse an Erlang form</fsummary>
       <desc>
         <p>Reads data from the standard input (<c><anno>IoDevice</anno></c>),
-          prompting it with <c><anno>Prompt</anno></c>. Starts reading at line number
-          <c><anno>StartLine</anno></c> (1). The data is tokenized and parsed as if
+          prompting it with <c><anno>Prompt</anno></c>. Starts reading at
+          line number <c><anno>StartLine</anno></c> (1). The argument
+          <c><anno>Options</anno></c> is passed on as the
+          <c>Options</c> argument of the <c>erl_scan:tokens/4</c>
+          function. The data is tokenized and parsed as if
           it were an Erlang form - one of the valid Erlang expressions
           in an Erlang source file - until a final '.' is reached. It
           returns:</p>