[stdlib, kernel] Introduce Unicode support for Erlang source files

Expect modifications, additions and corrections.

There is a kludge in file_io_server and
erl_scan:continuation_location() that's not so pleasing.

1 files changed, 63 insertions, 50 deletions

diff --git a/lib/stdlib/doc/src/unicode_usage.xml b/lib/stdlib/doc/src/unicode_usage.xml
index bbcd49a934..e93d49b24a 100644
--- a/lib/stdlib/doc/src/unicode_usage.xml
+++ b/lib/stdlib/doc/src/unicode_usage.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="latin1" ?>
+<?xml version="1.0" encoding="utf8" ?>
 <!DOCTYPE chapter SYSTEM "chapter.dtd">
 
 <chapter>
@@ -33,7 +33,7 @@
     <file>unicode_usage.xml</file>
   </header>
 <p>Implementing support for Unicode character sets is an ongoing process. The Erlang Enhancement Proposal (EEP) 10 outlines the basics of Unicode support and also specifies a default encoding in binaries that all Unicode-aware modules should handle in the future.</p>
-<p>The functionality described in EEP10 is implemented in Erlang/OTP as of R13A, but that's by no means the end of it. More functionality will be needed in the future and more OTP-libraries might need updating to cope with Unicode data. One example of future development is obvious when reading this manual, our documentation format is limited to the ISO-latin-1 character range, why no Unicode characters beyond that range will occur in this document.</p>
+<p>The functionality described in EEP10 is implemented in Erlang/OTP as of R13A, but that's by no means the end of it. More functionality will be needed in the future and more OTP-libraries might need updating to cope with Unicode data.</p>
 <p>This guide outlines the current Unicode support and gives a couple of recipes for working with Unicode data.</p>
 <section>
 <title>What Unicode is</title>
@@ -61,20 +61,20 @@
 <p>The standard list encoding for strings is therefore easily extendible to cope with the whole Unicode range: A Unicode string in Erlang is simply a list containing integers, each integer being a valid Unicode codepoint and representing one character in the Unicode character set.</p>
 <p>Regular Erlang strings in ISO-latin-1 are a subset of their Unicode strings.</p>
 
-<p>Binaries on the other hand are more troublesome. For performance reasons, programs often store textual data in binaries instead of lists, mainly because they are more compact (one byte per character instead of two words per character, as is the case with lists). Using erlang:list_to_binary/1, an regular Erlang string can be converted into a binary, effectively using the ISO-latin-1 encoding in the binary - one byte per character. This is very convenient for those regular Erlang strings, but cannot be done for Unicode lists.</p>
+<p>Binaries on the other hand are more troublesome. For performance reasons, programs often store textual data in binaries instead of lists, mainly because they are more compact (one byte per character instead of two words per character, as is the case with lists). Using erlang:list_to_binary/1, a regular Erlang string can be converted into a binary, effectively using the ISO-latin-1 encoding in the binary - one byte per character. This is very convenient for those regular Erlang strings, but cannot be done for Unicode lists.</p>
 <p>As the UTF-8 encoding is widely spread and provides the most compact storage, it is selected as the standard encoding of Unicode characters in binaries for Erlang.</p>
 <p>The standard binary encoding is used whenever a library function in Erlang should cope with Unicode data in binaries, but is of course not enforced when communicating externally. Functions and bit-syntax exist to encode and decode both UTF-8, UTF-16 and UTF-32 in binaries. Library functions dealing with binaries and Unicode in general, however, only deal with the default encoding.</p>
 
 <p>Character data may be combined from several sources, sometimes available in a mix of strings and binaries. Erlang has for long had the concept of iodata or iolists, where binaries and lists can be combined to represent a sequence of bytes. In the same way, the Unicode aware modules often allow for combinations of binaries and lists where the binaries have characters encoded in UTF-8 and the lists contain such binaries or numbers representing Unicode codepoints:</p>
 <code type="none">
 unicode_binary() = binary() with characters encoded in UTF-8 coding standard
-unicode_char() = integer() representing valid unicode codepoint
+unicode_char() = integer() >= 0 representing valid unicode codepoint
 
 chardata() = charlist() | unicode_binary()
 
 charlist() = [unicode_char() | unicode_binary() | charlist()]
   a unicode_binary is allowed as the tail of the list</code>
-<p>The module <c>unicode</c> in stdlib even supports similar mixes with binaries containing other encodings than UTF-8, but that is a special case to allow for conversions to and from external data:</p>
+<p>The module <c>unicode</c> in STDLIB even supports similar mixes with binaries containing other encodings than UTF-8, but that is a special case to allow for conversions to and from external data:</p>
     <code type="none">
 external_unicode_binary() = binary() with characters coded in a user specified Unicode 
   encoding other than UTF-8 (UTF-16 or UTF-32)
@@ -82,12 +82,18 @@ external_unicode_binary() = binary() with characters coded in a user specified U
 external_chardata() = external_charlist() | external_unicode_binary()
 
 external_charlist() = [unicode_char() | external_unicode_binary() | external_charlist()]
-  an external_unicode_binary is allowed as the tail of the list</code>
+  an external_unicode_binary() is allowed as the tail of the list</code>
 </section>
 <section>
 <title>Basic language support for Unicode</title>
-<p>First of all, Erlang is still defined to be written in the ISO-latin-1 character set. Functions have to be named in that character set, atoms are restricted to ISO-latin-1 and regular strings are still lists of characters 0..255 in the ISO-latin-1 encoding. This has not (yet) changed, but the language has been slightly extended to cope with Unicode characters and encodings.</p>
-
+<p><marker id="unicode_in_erlang"/>As of Erlang/OTP R16 Erlang can be
+written in ISO-latin-1 or Unicode (UTF-8). The details on how to state
+the encoding of an Erlang source file can be found in <seealso
+marker="stdlib:epp#encoding">epp(3)</seealso>. Strings and comments
+can be written using Unicode, but functions still have to be named in
+ISO-latin-1 and atoms are restricted to ISO-latin-1. Erlang/OTP R18 is
+expected to handle functions named in Unicode as well as Unicode
+atoms.</p>
 <section>
 <title>Bit-syntax</title>
 <p>The bit-syntax contains types for coping with binary data in the three main encodings. The types are named <c>utf8</c>, <c>utf16</c> and <c>utf32</c> respectively. The <c>utf16</c> and <c>utf32</c> types can be in a big- or little-endian variant:</p>
@@ -102,35 +108,19 @@ Bin4 = &lt;&lt;"Hello"/utf16&gt;&gt;,</code>
 </section>
 <section>
 <title>String- and character-literals</title>
-<warning>
-<p>The literal syntax described here may be subject to change in R13B, it has not yet passed the usual process for language changes approval.</p>
-</warning>
-<p>It is convenient to be able to write a list of Unicode characters in the string syntax. However, the language specifies strings as being in the ISO-latin-1 character set which the compiler tool chain as well as many other tools expect.</p>
-<p>Also the source code is (for now) still expected to be written using the ISO-latin-1 character set, why Unicode characters beyond that range cannot be entered in string literals.</p>
-<p>To make it easier to enter Unicode characters in the shell, it allows strings with Unicode characters on input, immediately converting them to regular lists of integers. They will, by the evaluator etc be viewed as if they were input using the regular list syntax, which is - in the end - how the language actually treats them. They will in the same way not be output as strings by i.e <c>io:write/2</c> or <c>io:format/3</c> unless the format string supplied to <c>io:format</c> uses the Unicode translation modifier (which we will talk about later).</p>
-<p>For source code, there is an extension to the \OOO (backslash followed by three octal numbers) and \xHH (backslash followed by 'x', followed by two hexadecimal characters) syntax, namely \x{H ...} (a backslash followed by an 'x', followed by left curly bracket, any number of hexadecimal digits and a terminating right curly bracket). This allows for entering characters of any codepoint literally in a string. The string is immediately converted into a list by the scanner however, which is obvious when calling it directly:</p>
-<pre>
-1> <input>erl_scan:string("\"X\".").</input>
-{ok,[{string,1,"X"},{dot,1}],1}
-2> <input>erl_scan:string("\"\x{400}\".").</input>
-{ok,[{'[',1},{integer,1,1024},{']',1},{dot,1}],1}</pre>
-<p>Character literals, or rather integers representing Unicode codepoints can be expressed in a similar way using $\x{H ...}:</p>
-<pre>
-4> <input>$\x{400}.</input>
-1024</pre>
-<p>This also is a translation by the scanner:</p>
-<pre>
-5> <input>erl_scan:string("$Y.").</input>
-{ok,[{char,1,89},{dot,1}],1}
-6> <input>erl_scan:string("$\x{400}.").</input>
-{ok,[{integer,1,1024},{dot,1}],1}</pre>
-<p>In the shell, if using a Unicode input device, '$' can be followed directly by a Unicode character producing an integer. In the following example, let's imagine the character 'c' is actually a Cyrillic 's' (looking fairly similar):</p>
+<p>For source code, there is an extension to the \OOO (backslash
+followed by three octal numbers) and \xHH (backslash followed by 'x',
+followed by two hexadecimal characters) syntax, namely \x{H ...} (a
+backslash followed by an 'x', followed by left curly bracket, any
+number of hexadecimal digits and a terminating right curly bracket).
+This allows for entering characters of any codepoint literally in a
+string even when the encoding is ISO-latin-1.</p>
+</section>
+<p>In the shell, if using a Unicode input device, '$' can be followed directly by a Unicode character producing an integer. In the following example the codepoint of a Cyrillic 's' is output:</p>
 <pre>
-7> <input>$c.</input>
+7> <input>$с.</input>
 1089</pre>
 </section>
-<p>The literal syntax allowing Unicode characters is to be viewed as "syntactic sugar", but is, as such, fairly useful.</p>
-</section> 
 <section>
 <title>The interactive shell</title>
 <p>The interactive Erlang shell, when started towards a terminal or started using the <c>werl</c> command on windows, can support Unicode input and output.</p>
@@ -147,25 +137,41 @@ en_US.UTF-8</pre>
 <p>To investigate what Erlang thinks about the terminal, the <c>io:getopts()</c> call can be used when the shell is started:</p>
 <pre>
 $ <input>LC_CTYPE=en_US.ISO-8859-1 erl</input>
-Erlang R13A (erts-5.7) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]
+Erlang R13A (erts-5.10) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]
 
-Eshell V5.7  (abort with ^G)
+Eshell V5.10  (abort with ^G)
 1> <input>lists:keyfind(encoding,1,io:getopts()).</input>
 {encoding,latin1}
 2> <input>q().</input>
 ok
 $ <input>LC_CTYPE=en_US.UTF-8 erl</input>
-Erlang R13A (erts-5.7) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]
+Erlang R13A (erts-5.10) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]
 
-Eshell V5.7  (abort with ^G)
+Eshell V5.10  (abort with ^G)
 1> <input>lists:keyfind(encoding,1,io:getopts()).</input>
 {encoding,unicode}
 2></pre>
 <p>When (finally?) everything is in order with the locale settings, fonts and the terminal emulator, you probably also have discovered a way to input characters in the script you desire. For testing, the simplest way is to add some keyboard mappings for other languages, usually done with some applet in your desktop environment. In my KDE environment, I start the KDE Control Center (Personal Settings), select "Regional and Accessibility" and then "Keyboard Layout". On Windows XP&reg;, I start Control Panel->Regional and Language Options, select the Language tab and click the Details... button in the square named "Text services and input Languages". Your environment probably provides similar means of changing the keyboard layout. Make sure you have a way to easily switch back and forth between keyboards if you are not used to this, entering commands using a Cyrillic character set is, as an example, not easily done in the Erlang shell.</p>
 <p>Now you are set up for some Unicode input and output. The simplest thing to do is of course to enter a string in the shell:</p>
-<image file="ushell1.gif"><icaption>Cyrillic characters in an Erlang shell</icaption></image>
+<pre>
+$ <input>erl</input>
+Eshell V5.10  (abort with ^G)
+1> <input>lists:keyfind(encoding, 1, io:getopts()).</input>
+{encoding,unicode}
+2> <input>"уницоде"</input>
+"уницоде"
+3> <input>io:format("~ts~n", [v(2)]).</input>
+уницоде
+ok
+4> </pre>
 <p>While strings can be input as Unicode characters, the language elements are still limited to the ISO-latin-1 character set. Only character constants and strings are allowed to be beyond that range:</p>
-<image file="ushell2.gif"><icaption>Unicode characters in allowed and disallowed context</icaption></image>
+<pre>
+Eshell V5.10  (abort with ^G)
+1> <input>$ξ</input>
+958
+2> <input>уницоде.</input>
+* 1: illegal character
+2> </pre>
 </section> 
 <section>
 <title>Unicode file names</title>
@@ -183,7 +189,7 @@ Eshell V5.7  (abort with ^G)
 <p>A raw file name is not a list, but a binary. Many non core applications still do not handle file names given as binaries, why such raw names are avoided by default. This means that systems having implemented Unicode file naming through transparent file systems and an UTF-8 convention, do not by default have Unicode file naming turned on. Explicitly turning Unicode file name handling on for these types of systems is considered experimental.</p>
 </item>
 </taglist>
-<p>The Unicode file naming support was introduced with OTP release R14B01. A VM operating in Unicode file mode can work with files having names in any language or character set (as long as it's supported by the underlying OS and file system). The Unicode character list is used to denote file or directory names and if the file system content is listed, you will also be able to get Unicode lists as return value. The support lies in the kernel and stdlib modules, why most applications (that does not explicitly require the file names to be in the ISO-latin-1 range) will benefit from the Unicode support without change.</p>
+<p>The Unicode file naming support was introduced with OTP release R14B01. A VM operating in Unicode file mode can work with files having names in any language or character set (as long as it's supported by the underlying OS and file system). The Unicode character list is used to denote file or directory names and if the file system content is listed, you will also be able to get Unicode lists as return value. The support lies in the Kernel and STDLIB modules, why most applications (that does not explicitly require the file names to be in the ISO-latin-1 range) will benefit from the Unicode support without change.</p>
 
 <p>On Operating systems with mandatory Unicode file names, this means that you more easily conform to the file names of other (non Erlang) applications, and you can also process file names that, at least on Windows, were completely inaccessible (due to having names that could not be represented in ISO-latin-1). Also you will avoid creating incomprehensible file names on MacOSX as the vfs layer of the OS will accept all your file names as UTF-8 and will not rewrite them.</p>
 
@@ -197,8 +203,9 @@ Eshell V5.7  (abort with ^G)
 
 <section>
 <title>Notes about raw file names and automatic file name conversion</title>
-<p>Raw file names is introduced together with Unicode file name support in erts-5.8.2 (OTP R14B01). The reason &quot;raw file names&quot; is introduced in the system is to be able to consistently represent file names given in different encodings on the same system. Having the VM automatically translate a file name that is not in UTF-8 to a list of Unicode characters might seem practical, but this would open up for both duplicate file names and other inconsistent behavior. Consider a directory containing a file named &quot;bj�rn&quot; in ISO-latin-1, while the Erlang VM is operating in Unicode file name mode (and therefore expecting UTF-8 file naming). The ISO-latin-1 name is not valid UTF-8 and one could be tempted to think that automatic conversion in for example <c>file:list_dir/1</c> is a good idea. But what would happen if we later tried to open the file and have the name as a Unicode list (magically converted from the ISO-latin-1 file name)? The VM will convert the file name given to UTF-8, as this is the encoding expected. Effectively this means trying to open the file named &lt;&lt;&quot;bj�rn&quot;/utf8&gt;&gt;. This file does not exist, and even if it existed it would not be the same file as the one that was listed. We could even create two files named &quot;bj�rn&quot;, one named in the UTF-8 encoding and one not. If <c>file:list_dir/1</c> would automatically convert the ISO-latin-1 file name to a list, we would get two identical file names as the result. To avoid this, we need to differentiate between file names being properly encoded according to the Unicode file naming convention (i.e. UTF-8) and file names being invalid under the encoding. This is done by representing invalid encoding as &quot;raw&quot; file names, i.e. as binaries.</p>
-<p>The core system of Erlang (kernel and stdlib) accepts raw file names except for loadable drivers and executables invoked using <c>open_port({spawn, ...} ...)</c>. <c>open_port({spawn_executable, ...} ...)</c> however does accept them. As mentioned earlier, the arguments given in the option list to <c>open_port({spawn_executable, ...} ...)</c> undergo the same conversion as the file names, meaning that the executable will be provided with arguments in UTF-8 as well. This translation is avoided consistently with how the file names are treated, by giving the argument as a binary.</p>
+
+<p>Raw file names is introduced together with Unicode file name support in erts-5.8.2 (OTP R14B01). The reason &quot;raw file names&quot; is introduced in the system is to be able to consistently represent file names given in different encodings on the same system. Having the VM automatically translate a file name that is not in UTF-8 to a list of Unicode characters might seem practical, but this would open up for both duplicate file names and other inconsistent behavior. Consider a directory containing a file named &quot;björn&quot; in ISO-latin-1, while the Erlang VM is operating in Unicode file name mode (and therefore expecting UTF-8 file naming). The ISO-latin-1 name is not valid UTF-8 and one could be tempted to think that automatic conversion in for example <c>file:list_dir/1</c> is a good idea. But what would happen if we later tried to open the file and have the name as a Unicode list (magically converted from the ISO-latin-1 file name)? The VM will convert the file name given to UTF-8, as this is the encoding expected. Effectively this means trying to open the file named &lt;&lt;&quot;björn&quot;/utf8&gt;&gt;. This file does not exist, and even if it existed it would not be the same file as the one that was listed. We could even create two files named &quot;björn&quot;, one named in the UTF-8 encoding and one not. If <c>file:list_dir/1</c> would automatically convert the ISO-latin-1 file name to a list, we would get two identical file names as the result. To avoid this, we need to differentiate between file names being properly encoded according to the Unicode file naming convention (i.e. UTF-8) and file names being invalid under the encoding. This is done by representing invalid encoding as &quot;raw&quot; file names, i.e. as binaries.</p>
+<p>The core system of Erlang (Kernel and STDLIB) accepts raw file names except for loadable drivers and executables invoked using <c>open_port({spawn, ...} ...)</c>. <c>open_port({spawn_executable, ...} ...)</c> however does accept them. As mentioned earlier, the arguments given in the option list to <c>open_port({spawn_executable, ...} ...)</c> undergo the same conversion as the file names, meaning that the executable will be provided with arguments in UTF-8 as well. This translation is avoided consistently with how the file names are treated, by giving the argument as a binary.</p>
 <p>To force Unicode file name translation mode on systems where this is not the default is considered experimental in OTP R14B01 due to the raw file names possibly being a new experience to the programmer and that the non core applications of OTP are not tested for compliance with raw file names yet. Unicode file name translation is expected to be default in future releases.</p>
 <p>If working with raw file names, one can still conform to the encoding convention of the Erlang VM by using the <c>file:native_name_encoding/0</c> function, which returns either the atom <c>latin1</c> or the atom <c>utf8</c> depending on the file name translation mode. On Linux, a VM started without explicitly stating the file name translation mode will default to <c>latin1</c> as the native file name encoding, why file names on the disk encoded as UTF-8 will be returned as a list of the names interpreted as ISO-latin-1. The &quot;UTF-8 list&quot; is not a practical type for displaying or operating on in Erlang, but it is backward compatible and usable in all functions requiring a file name. On Windows and MacOSX, the default behavior is that of file name translation, why the <c>file:native_name_encoding/0</c> by default returns <c>utf8</c> on those systems (the fact that Windows actually does not use UTF-8 on the file system level can safely be ignored by the Erlang programmer). The default behavior can be changed using the <c>+fnu</c> or <c>+fnl</c> options to the VM, see the <c>erl</c> command manual page.</p> 
 <p>Even if you are operating without Unicode file naming translation automatically done by the VM, you can access and create files with names in UTF-8 encoding by using raw file names encoded as UTF-8. Enforcing the UTF-8 encoding regardless of the mode the Erlang VM is started in might, in some circumstances be a good idea, as the convention of using UTF-8 file names is spreading.</p>  
@@ -206,7 +213,7 @@ Eshell V5.7  (abort with ^G)
 <section>
 <title>Notes about MacOSX</title>
 <p>MacOSXs vfs layer enforces UTF-8 file names in a quite aggressive way. Older versions did this by simply refusing to create non UTF-8 conforming file names, while newer versions replace offending bytes with the sequence &quot;%HH&quot;, where HH is the original character in hexadecimal notation. As Unicode translation is enabled by default on MacOSX, the only way to come up against this is to either start the VM with the <c>+fnl</c> flag or to use a raw file name in <c>latin1</c> encoding. In that case, the file can not be opened with the same name as the one used to create this. The problem is by design in newer versions of MacOSX.</p>
-<p>MacOSX also reorganizes the names of files so that the representation of accents etc is denormalized, i.e. the character <c>�</c> is represented as the codepoints [111,776], where 111 is the character <c>o</c> and 776 is a special accent character. This type of denormalized Unicode is otherwise very seldom used and Erlang normalizes those file names on retrieval, so that denormalized file names is not passed up to the Erlang application. In Erlang the file name &quot;bj�rn&quot; is retrieved as [98,106,246,114,110], not as [98,106,117,776,114,110], even though the file system might think differently.</p>
+<p>MacOSX also reorganizes the names of files so that the representation of accents etc is denormalized, i.e. the character <c>ö</c> is represented as the codepoints [111,776], where 111 is the character <c>o</c> and 776 is a special accent character. This type of denormalized Unicode is otherwise very seldom used and Erlang normalizes those file names on retrieval, so that denormalized file names is not passed up to the Erlang application. In Erlang the file name &quot;björn&quot; is retrieved as [98,106,246,114,110], not as [98,106,117,776,114,110], even though the file system might think differently.</p>
 </section>
 </section>
 <section>
@@ -276,17 +283,23 @@ open_bom_file_for_writing(File,Encoding) -&gt;
 <title>Formatted input and output</title>
 <p>When reading and writing to Unicode-aware entities, like the User or a file opened for Unicode translation, you will probably want to format text strings using the functions in <seealso marker="stdlib:io">io</seealso> or <seealso marker="stdlib:io_lib">io_lib</seealso>. For backward compatibility reasons, these functions don't accept just any list as a string, but require e special "translation modifier" when working with Unicode texts. The modifier is "t". When applied to the "s" control character in a formatting string, it accepts all Unicode codepoints and expect binaries to be in UTF-8:</p>
 <pre>
-1> <input>io:format("~ts~n",[&lt;&lt;"���"/utf8&gt;&gt;]).</input>
-���
-ok
-2> <input>io:format("~s~n",[&lt;&lt;"���"/utf8&gt;&gt;]).</input>
+1> <input>io:format("~ts~n",[&lt;&lt;"åäö"/utf8&gt;&gt;]).</input>
 åäö
+ok
+2> <input>io:format("~s~n",[&lt;&lt;"åäö"/utf8&gt;&gt;]).</input>
+åäö
 ok</pre>
 <p>Obviously the second <c>io:format</c> gives undesired output because the UTF-8 binary is not in latin1. Because ISO-latin-1 is still the defined character set of Erlang, the non prefixed "s" control character expects ISO-latin-1 in binaries as well as lists.</p>
 <p>As long as the data is always lists, the "t" modifier can be used for any string, but when binary data is involved, care must be taken to make the tight choice of formatting characters.</p>
 <p>The function <c>format</c> in <c>io_lib</c> behaves similarly. This function is defined to return a deep list of characters and the output could easily be converted to binary data for outputting on a device of any kind by a simple <c>erlang:list_to_binary</c>. When the translation modifier is used, the list can however contain characters that cannot be stored in one byte. The call to <c>erlang:list_to_binary</c> will in that case fail. However, if the io_server you want to communicate with is Unicode-aware, the list returned can still be used directly:</p>
-<image file="ushell3.gif"><icaption>io_lib:format with Unicode translation</icaption></image>
-<p>The Unicode string is returned as a Unicode list, why the return value of <c>io_lib:format</c> no longer qualifies as a regular Erlang string (the function <seealso marker="stdlib:io_lib#deep_char_list/1">io_lib:deep_char_list</seealso> will, as an example, return <c>false</c>). The Unicode list is however valid input to the <seealso marker="stdlib:io#put_chars/2">io:put_chars</seealso> function, so data can be output on any Unicode capable device anyway. If the device is a terminal, characters will be output in the \x{H ...} format if encoding is <c>latin1</c> otherwise in UTF-8 (for the non-interactive terminal - "oldshell" or "noshell") or whatever is suitable to show the character properly (for an interactive terminal - the regular shell). The bottom line is that you can always send Unicode data to the <c>standard_io</c> device. Files will however only accept Unicode codepoints beyond ISO-latin-1 if <c>encoding</c> is set to something else than <c>latin1</c>.</p>
+<pre>
+Eshell V5.10 (abort with ^G)
+1> <input>io_lib:format("~ts~n", ["θνιψοδε"]).</input>
+["θνιψοδε","\n"]
+2> <input>io:put_chars(io_lib:format("~ts~n", ["θνιψοδε"])).</input>
+θνιψοδε
+ok</pre>
+<p>The Unicode string is returned as a Unicode list, which is recognized as such since the Erlang shell uses the Unicode encoding. The Unicode list is valid input to the <seealso marker="stdlib:io#put_chars/2">io:put_chars</seealso> function, so data can be output on any Unicode capable device. If the device is a terminal, characters will be output in the \x{H ...} format if encoding is <c>latin1</c> otherwise in UTF-8 (for the non-interactive terminal - "oldshell" or "noshell") or whatever is suitable to show the character properly (for an interactive terminal - the regular shell). The bottom line is that you can always send Unicode data to the <c>standard_io</c> device. Files will however only accept Unicode codepoints beyond ISO-latin-1 if <c>encoding</c> is set to something else than <c>latin1</c>.</p>
 </section>
 <section>
 <title>Heuristic identification of UTF-8</title>