diff options
author | Loïc Hoguin <[email protected]> | 2016-03-28 15:36:42 +0200 |
---|---|---|
committer | Loïc Hoguin <[email protected]> | 2016-03-28 15:36:42 +0200 |
commit | fe3492a98de29942477b061cd02c92246f4bf85a (patch) | |
tree | 2255b796a657e6e4dfb72beec1141258d17f1220 /articles/index.xml | |
download | ninenines.eu-fe3492a98de29942477b061cd02c92246f4bf85a.tar.gz ninenines.eu-fe3492a98de29942477b061cd02c92246f4bf85a.tar.bz2 ninenines.eu-fe3492a98de29942477b061cd02c92246f4bf85a.zip |
Initial commit, new website system
Diffstat (limited to 'articles/index.xml')
-rw-r--r-- | articles/index.xml | 1919 |
1 files changed, 1919 insertions, 0 deletions
diff --git a/articles/index.xml b/articles/index.xml new file mode 100644 index 00000000..81663a56 --- /dev/null +++ b/articles/index.xml @@ -0,0 +1,1919 @@ +<?xml version="1.0" encoding="utf-8" standalone="yes" ?> +<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"> + <channel> + <title>Articles on Nine Nines</title> + <link>http://ninenines.eu/articles/</link> + <description>Recent content in Articles on Nine Nines</description> + <generator>Hugo -- gohugo.io</generator> + <language>en-us</language> + <lastBuildDate>Wed, 02 Sep 2015 00:00:00 +0100</lastBuildDate> + <atom:link href="http://ninenines.eu/articles/index.xml" rel="self" type="application/rss+xml" /> + + <item> + <title>The Erlanger Playbook September 2015 Update</title> + <link>http://ninenines.eu/articles/erlanger-playbook-september-2015-update/</link> + <pubDate>Wed, 02 Sep 2015 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/erlanger-playbook-september-2015-update/</guid> + <description><div class="paragraph"><p>An update to The Erlanger Playbook is now available!</p></div>
+<div class="paragraph"><p>The Erlanger Playbook is a book about software development using
+Erlang. It currently covers all areas from the conception, design,
+the writing of code, documentation and tests.</p></div>
+<div class="paragraph"><p>The book is still a work in progress. Future topics will include
+refactoring, debugging and tracing, benchmarking, releases, community
+management (for open source projects).</p></div>
+<div class="paragraph"><p>This update fixes a number of things and adds two chapters: IOlists
+and Erlang building blocks.</p></div>
+<div class="paragraph"><p>Learn more about <a href="http://ninenines.eu/articles/erlanger-playbook">The Erlanger Playbook</a>!</p></div>
+<div class="paragraph"><p>This is a self-published ebook. The base price is 50€. All proceeds
+will be used to allow me to work on open source full time.</p></div>
+<div class="paragraph"><p>Thank you for helping me helping you help us all!</p></div>
+</description> + </item> + + <item> + <title>The Erlanger Playbook</title> + <link>http://ninenines.eu/articles/erlanger-playbook/</link> + <pubDate>Thu, 18 Jun 2015 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/erlanger-playbook/</guid> + <description><div class="paragraph"><p>I am proud to announce the pre-release of The Erlanger Playbook.</p></div>
+<div class="paragraph"><p>The Erlanger Playbook is a book about software development using
+Erlang. It currently covers all areas from the conception, design,
+the writing of code, documentation and tests.</p></div>
+<div class="paragraph"><p>The book is still a work in progress. Future topics will include
+refactoring, debugging and tracing, benchmarking, releases, community
+management (for open source projects).</p></div>
+<div class="paragraph"><p>The following sections are currently available:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+About this book; Future additions
+</p>
+</li>
+<li>
+<p>
+<em>Workflow:</em> Think; Write; Stay productive
+</p>
+</li>
+<li>
+<p>
+<em>Documentation:</em> On documentation; Tutorials; User guide; Manual
+</p>
+</li>
+<li>
+<p>
+<em>Code:</em> Starting a project; Version control; Project structure; Code style; Best practices; Special processes
+</p>
+</li>
+<li>
+<p>
+<em>Tests:</em> On testing; Success typing analysis; Manual testing; Unit testing; Functional testing
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>Read a preview: <a href="http://ninenines.eu/res/erlanger-preview.pdf">Special processes</a></p></div>
+<div class="paragraph"><p>The book is currently just shy of 100 pages. The final version
+of the book is planned to be between 200 and 250 pages.
+A print version of the book will be considered once the final
+version gets released. The printed book is <strong>not</strong> included
+in the price.</p></div>
+<div class="paragraph"><p>This is a self-published book. The base price is 50€. All proceeds
+will be used to allow me to work on open source full time.</p></div>
+<form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top">
+<input type="hidden" name="cmd" value="_s-xclick">
+<input type="hidden" name="hosted_button_id" value="9M44HJCGX3GVN">
+<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
+<img alt="" border="0" src="https://www.paypalobjects.com/fr_FR/i/scr/pixel.gif" width="1" height="1">
+</form>
+<div class="paragraph"><p>You are more than welcome to pay extra by using this second button.
+It allows you to set the price you want. Make sure to set it to at least
+50€ to receive the book.</p></div>
+<form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top">
+<input type="hidden" name="cmd" value="_s-xclick">
+<input type="hidden" name="hosted_button_id" value="BBW9TR9LBK8C2">
+<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
+<img alt="" border="0" src="https://www.paypalobjects.com/fr_FR/i/scr/pixel.gif" width="1" height="1">
+</form>
+<div class="paragraph"><p>Make sure to provide a valid email address.</p></div>
+<div class="paragraph"><p>There will be a <strong>delay</strong> between payment and sending of the book.
+This process is currently manual.</p></div>
+<div class="paragraph"><p>As the book is a pre-release, feedback is more than welcome. You can
+send your comments to erlanger@ this website.</p></div>
+<div class="paragraph"><p>The plan is to add about 20 pages every month until it is completed.
+You will receive updates to the book for free as soon as they are available.</p></div>
+<div class="paragraph"><p>Huge thanks for your interest in buying this book!</p></div>
+</description> + </item> + + <item> + <title>Validating UTF-8 binaries with Erlang</title> + <link>http://ninenines.eu/articles/erlang-validate-utf8/</link> + <pubDate>Fri, 06 Mar 2015 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/erlang-validate-utf8/</guid> + <description><div class="paragraph"><p>Yesterday I pushed Websocket permessage-deflate to
+Cowboy master. I also pushed
+<a href="https://github.com/ninenines/cowlib/commit/7e4983b70ddf8cedb967e36fba6a600731bdad5d">a
+change in the way the code validates UTF-8 data</a>
+(required for text and close frames as per the spec).</p></div>
+<div class="paragraph"><p>When looking into why the permessage-deflate tests
+in autobahntestsuite were taking such a long time, I
+found that autobahn is using an adaptation of the
+algorithm named &lt;a href="http://bjoern.hoehrmann.de/utf-8/decoder/dfa/"&gt;Flexible
+and Economical UTF-8 Decoder&lt;/a&gt;. This is the C99
+implementation:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-style: italic"><span style="color: #9A1900">// Copyright (c) 2008-2009 Bjoern Hoehrmann &lt;[email protected]&gt;</span></span>
+<span style="font-style: italic"><span style="color: #9A1900">// See http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.</span></span>
+
+<span style="font-weight: bold"><span style="color: #000080">#define</span></span> UTF8_ACCEPT <span style="color: #993399">0</span>
+<span style="font-weight: bold"><span style="color: #000080">#define</span></span> UTF8_REJECT <span style="color: #993399">1</span>
+
+<span style="font-weight: bold"><span style="color: #0000FF">static</span></span> <span style="font-weight: bold"><span style="color: #0000FF">const</span></span> <span style="color: #008080">uint8_t</span> utf8d<span style="color: #990000">[]</span> <span style="color: #990000">=</span> <span style="color: #FF0000">{</span>
+ <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 00..1f</span></span>
+ <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 20..3f</span></span>
+ <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 40..5f</span></span>
+ <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 60..7f</span></span>
+ <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 80..9f</span></span>
+ <span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// a0..bf</span></span>
+ <span style="color: #993399">8</span><span style="color: #990000">,</span><span style="color: #993399">8</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// c0..df</span></span>
+ <span style="color: #993399">0xa</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x4</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// e0..ef</span></span>
+ <span style="color: #993399">0xb</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x5</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// f0..ff</span></span>
+ <span style="color: #993399">0x0</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x2</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x5</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x7</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x4</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s0..s0</span></span>
+ <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s1..s2</span></span>
+ <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s3..s4</span></span>
+ <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s5..s6</span></span>
+ <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s7..s8</span></span>
+<span style="color: #FF0000">}</span><span style="color: #990000">;</span>
+
+uint32_t inline
+<span style="font-weight: bold"><span style="color: #000000">decode</span></span><span style="color: #990000">(</span>uint32_t<span style="color: #990000">*</span> state<span style="color: #990000">,</span> uint32_t<span style="color: #990000">*</span> codep<span style="color: #990000">,</span> <span style="color: #008080">uint32_t</span> byte<span style="color: #990000">)</span> <span style="color: #FF0000">{</span>
+ <span style="color: #008080">uint32_t</span> type <span style="color: #990000">=</span> utf8d<span style="color: #990000">[</span>byte<span style="color: #990000">];</span>
+
+ <span style="color: #990000">*</span>codep <span style="color: #990000">=</span> <span style="color: #990000">(*</span>state <span style="color: #990000">!=</span> UTF8_ACCEPT<span style="color: #990000">)</span> <span style="color: #990000">?</span>
+ <span style="color: #990000">(</span>byte <span style="color: #990000">&amp;</span> <span style="color: #993399">0x3fu</span><span style="color: #990000">)</span> <span style="color: #990000">|</span> <span style="color: #990000">(*</span>codep <span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">6</span><span style="color: #990000">)</span> <span style="color: #990000">:</span>
+ <span style="color: #990000">(</span><span style="color: #993399">0xff</span> <span style="color: #990000">&gt;&gt;</span> type<span style="color: #990000">)</span> <span style="color: #990000">&amp;</span> <span style="color: #990000">(</span>byte<span style="color: #990000">);</span>
+
+ <span style="color: #990000">*</span>state <span style="color: #990000">=</span> utf8d<span style="color: #990000">[</span><span style="color: #993399">256</span> <span style="color: #990000">+</span> <span style="color: #990000">*</span>state<span style="color: #990000">*</span><span style="color: #993399">16</span> <span style="color: #990000">+</span> type<span style="color: #990000">];</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">return</span></span> <span style="color: #990000">*</span>state<span style="color: #990000">;</span>
+<span style="color: #FF0000">}</span></tt></pre></div></div>
+<div class="paragraph"><p>And this is the Erlang implementation I came up with:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-style: italic"><span style="color: #9A1900">%% This function returns 0 on success, 1 on error, and 2..8 on incomplete data.</span></span>
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;&gt;&gt;</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span> <span style="color: #009900">State</span>;
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">128</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">8</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">4</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">194</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">224</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">224</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">4</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">225</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">237</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">237</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">5</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">238</span>; <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">239</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">240</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">6</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">241</span>; <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">242</span>; <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">243</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">7</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">244</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">8</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">_</span>, <span style="color: #990000">_</span>) <span style="color: #990000">-&gt;</span> <span style="color: #993399">1</span><span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>Does it look similar to you? So how did we get there?</p></div>
+<div class="paragraph"><p>I started with a naive implementation of the original. First, we
+don&#8217;t need the codepoint calculated and extracted for our validation
+function. We just want to know the data is valid, so we only need to
+calculate the next state. Then, the only thing we needed to be careful
+about was that tuples are 1-based, and that we need to stop processing
+the binary when we get the state 1 or when the binary is empty.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;&gt;&gt;</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span> <span style="color: #009900">State</span>;
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">_</span>, <span style="color: #993399">1</span>) <span style="color: #990000">-&gt;</span> <span style="color: #993399">1</span>;
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">257</span> <span style="color: #990000">+</span> <span style="color: #009900">State</span> <span style="color: #990000">*</span> <span style="color: #993399">16</span> <span style="color: #990000">+</span> <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">1</span> <span style="color: #990000">+</span> <span style="color: #009900">C</span>, <span style="font-weight: bold"><span style="color: #000080">?UTF8D</span></span>), <span style="font-weight: bold"><span style="color: #000080">?UTF8D</span></span>))<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>The macro <code>?UTF8D</code> is the tuple equivalent of the C array
+in the original code.</p></div>
+<div class="paragraph"><p>Compared to our previous algorithm, this performed about the same.
+In some situations a little faster, in some a little slower. In other words,
+not good enough. But because this new algorithm allows us to avoid a binary
+concatenation this warranted looking further.</p></div>
+<div class="paragraph"><p>It was time to step into crazy land.</p></div>
+<div class="paragraph"><p>Erlang is very good at pattern matching, even more so than doing some
+arithmetic coupled by fetching elements from a tuple. So I decided I was
+going to write all possible clauses for all combinations of <code>C</code>
+and <code>State</code>. And by write I mean generate.</p></div>
+<div class="paragraph"><p>So I opened my Erlang shell, defined the variable <code>D</code> to be
+the tuple <code>?UTF8D</code> with its 400 elements, and then ran the
+following expression (after a bit of trial and error):</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #993399">16</span><span style="color: #990000">&gt;</span> <span style="font-weight: bold"><span style="color: #000000">file:write_file</span></span>(<span style="color: #FF0000">"out.txt"</span>,
+ [<span style="font-weight: bold"><span style="color: #000000">io_lib:format</span></span>(<span style="color: #FF0000">"validate_utf8(&lt;&lt; ~p, Rest/bits &gt;&gt;, ~p) -&gt; ~p;~n"</span>,
+ [<span style="color: #009900">C</span>, <span style="color: #009900">S</span>, <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">257</span> <span style="color: #990000">+</span> <span style="color: #009900">S</span> <span style="color: #990000">*</span> <span style="color: #993399">16</span> <span style="color: #990000">+</span> <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">1</span> <span style="color: #990000">+</span> <span style="color: #009900">C</span>, <span style="color: #009900">D</span>), <span style="color: #009900">D</span>)])
+ || <span style="color: #009900">C</span> <span style="color: #990000">&lt;-</span> <span style="font-weight: bold"><span style="color: #000000">lists:seq</span></span>(<span style="color: #993399">0</span>,<span style="color: #993399">255</span>), <span style="color: #009900">S</span> <span style="color: #990000">&lt;-</span> <span style="font-weight: bold"><span style="color: #000000">lists:seq</span></span>(<span style="color: #993399">0</span>,<span style="color: #993399">8</span>)])<span style="color: #990000">.</span>
+<span style="color: #FF6600">ok</span></tt></pre></div></div>
+<div class="paragraph"><p>The result is a 2304 lines long file, containing 2304 clauses.
+People who pay attention to what I say on Twitter will remember
+I said something around 3000 clauses, but that was just me not
+using the right number of states in my estimate.</p></div>
+<div class="paragraph"><p>There was a little more work to be done on this generated
+code that I did using regular expressions. We need to recurse
+when the resulting state is not 1. We also need to stop when
+the binary is empty, making it the 2305th clause.</p></div>
+<div class="paragraph"><p>Still, 2305 is a lot. But hey, the code did work, and faster
+than the previous implementation too! But hey, perhaps I could
+find a way to reduce its size.</p></div>
+<div class="paragraph"><p>Removing all the clauses that return 1 and putting a catch-all
+clause at the end instead reduced the number to about 500, and
+showed that many clauses were similar:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">0</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">1</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">2</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">3</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">4</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">5</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">6</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">7</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);</tt></pre></div></div>
+<div class="paragraph"><p>But also:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);</tt></pre></div></div>
+<div class="paragraph"><p>Patterns, my favorites!</p></div>
+<div class="paragraph"><p>A little more time was spent to edit the 500 or so clauses into
+smaller equivalents, testing that performance was not impacted, and
+comitting the result.</p></div>
+<div class="paragraph"><p>The patterns above can be found here in the resulting function:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">128</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="color: #990000">...</span>
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);
+<span style="color: #990000">...</span></tt></pre></div></div>
+<div class="paragraph"><p>I hope you enjoyed this post.</p></div>
+</description> + </item> + + <item> + <title>On open source</title> + <link>http://ninenines.eu/articles/on-open-source/</link> + <pubDate>Fri, 05 Sep 2014 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/on-open-source/</guid> + <description><div class="paragraph"><p>Last week I read a great article
+<a href="http://videlalvaro.github.io/2014/08/on-contributing-to-opensource.html">on
+contributing to open source</a> by Alvaro Videla. He makes
+many great points and I am in agreement with most of it.
+This made me want to properly explain my point of view with
+regard to open source and contributions. Unlike most open
+source evangelism articles I will not talk about ideals or
+any of that crap, but rather my personal feelings and
+experience.</p></div>
+<div class="paragraph"><p>I have been doing open source work for quite some time.
+My very first open source project was a graphics driver
+for (the very early version of) the PCSX2 emulator. That
+was more than ten years ago, and there
+<a href="http://ngemu.com/threads/gstaris-0-6.30469/">isn&#8217;t
+much left to look at today</a>. This was followed by a
+<a href="https://github.com/extend/wee">PHP framework</a>
+(started long before Zend Framework was even a thing) and
+a few other small projects. None of them really took off.
+It&#8217;s alright, that&#8217;s pretty much the fate of most open
+source projects. You spend a lot of work and sweat and
+get very little in return from others.</p></div>
+<div class="paragraph"><p>This sounds harsh but this is the reality of all open
+source projects. If you are thinking of building a project
+and releasing it as open source, you should be prepared
+for that. This is how most of your projects will feel like.
+Don&#8217;t release a project as open source thinking everyone
+will pat you on the back and cheer, this won&#8217;t happen. In
+fact if your project is a too small improvement over existing
+software, what many people will do is say you have NIH
+syndrome, regardless of the improvement you bring. So you
+need not to rely on other people in order to get your
+enjoyment out of building open source software.</p></div>
+<div class="paragraph"><p>In my case I get enjoyment from thinking about problems
+that need solving. Often times the problems are already
+solved, but nevermind that, I still think about them and
+sometimes come up with something I feel is better and then
+write code for it. Writing code is also fun, but not as
+fun as using my brain to imagine solutions.</p></div>
+<div class="paragraph"><p>You don&#8217;t need thousands of users to do that. So are
+users worthless to me then? No, of course not. In fact
+they are an important component: they bring me problems
+that need solving. So users are very important to me.
+But that&#8217;s not the only reason.</p></div>
+<div class="paragraph"><p>I got lucky that the Cowboy project became popular.
+And seeing it be this popular, and some of my other projects
+also do quite well, made me believe I could perhaps work
+full time on open source. If I can work full time then
+I can produce better software. What I had one hour to
+work on before I can now spend a day on, and experiment
+until I am satisfied. This is very useful because that
+means I can get it almost right from the beginning, and
+avoid the million API breaking changes that occured
+before Cowboy 1.0 was released.</p></div>
+<div class="paragraph"><p>To be able to work full time on open source however,
+I need money. This is a largely unspoken topic of open
+source work. The work is never free. You can download the
+product for free, but someone has to pay for the work
+itself. Life is unfortunately not free.</p></div>
+<div class="paragraph"><p>Large projects and some lucky people have their work
+sponsored by their employers. Everyone else has to deal
+with it differently. In my case I was sponsored for a
+while by the <a href="http://leo-project.net/leofs/">LeoFS</a>
+project, but that ended. I also had the Farwest fundraiser,
+which was a success, although the project stalled after that.
+(Fear not, as Farwest will make a comeback as a conglomerate
+of Web development projects in the future.) After that I set
+up the <a href="http://ninenines.eu/support/">sponsoring scheme</a>,
+which I can proudly say today brings in enough money to
+cover my food and shelter. Great!</p></div>
+<div class="paragraph"><p>This is a start, but it&#8217;s of course not enough. Life
+is a little more than food and shelter, and so I am still
+looking for sponsors. This is not a very glorious experience,
+as I am essentially looking for scraps that companies can
+throw away. Still, if a handful more companies were doing
+that, not only would I be able to live comfortably, but I
+would also be able to stop worrying about the future as I
+could put money on the side for when it gets rough.</p></div>
+<div class="paragraph"><p>A few companies giving me some scrap money so I could
+live and work independently is by far the most important
+thing anyone can do to help my projects, including Cowboy.
+Yes, they&#8217;re even more important than code contributions,
+bug reports and feedback. Because this money gives me the
+time I need to handle the code contributions, bug reports
+and feedback.</p></div>
+<div class="paragraph"><p>If Cowboy or another project is a large part of your
+product or infrastructure, then the best thing you can do
+is become a sponsor. The second best is opening tickets
+and/or providing feedback. The third best is providing
+good code contributions.</p></div>
+<div class="paragraph"><p>I will not expand on the feedback part. Feedback is
+very important, and even just a high five or a retweet
+is already good feedback. It&#8217;s not very complicated.</p></div>
+<div class="paragraph"><p>I want to expand a little on code contributions
+however. Not long ago I ran across the term "patch bomb"
+which means dropping patches and expecting the project
+maintainers to merge them and maintain them. I receive
+a lot of patches, and often have to refuse them. Causes
+for refusal vary. Some patches only benefit the people
+who submitted them (or a very small number of people).
+Some patches are not refined enough to be included.
+Others are out of scope of the project. These are some
+of the reasons why I refuse patches. Having limited
+time and resources, I have to focus my efforts on the
+code used by the larger number of users. I have to
+prioritize patches from submitters who are reactive
+and address the issues pointed out. And I have to plainly
+refuse other patches.</p></div>
+<div class="paragraph"><p>I believe this wraps up my thoughts on open source.
+Overall I had a great experience, the Erlang community
+being nice and understanding of the issues at hand in
+general. And if the money problem could be solved soon,
+then I would be one of the luckiest and happiest open
+source developer on Earth.</p></div>
+<div class="paragraph"><p>Think about it the next time you see a donation button
+or a request for funds or sponsoring. You can considerably
+improve an open source developer&#8217;s life with very little
+of your company&#8217;s money.</p></div>
+</description> + </item> + + <item> + <title>The story so far</title> + <link>http://ninenines.eu/articles/the-story-so-far/</link> + <pubDate>Sat, 23 Aug 2014 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/the-story-so-far/</guid> + <description><div class="paragraph"><p>As I am away from home with little to do (some call this
+a vacation) I wanted to reflect a little on the story so far,
+or how I arrived to Erlang and got to where I am now. The
+raw personal experience. It&#8217;ll be an article that&#8217;s more
+about social aspect, communities and marketing a project than
+technical considerations. As a period piece, it will also
+allow me to reflect on the evolution of Erlang in recent
+years.</p></div>
+<div class="paragraph"><p>Once upon a time-- Okay this isn&#8217;t a fairy tale. The story
+begins with a short chapter in 2010. The year 2010 started
+with a fairly major event in my life: the US servers for the
+online game I stopped playing a few months before, but was
+still involved with through its community, were closing. OMG!
+Someone found a way to log packets and started working on a
+private server; meanwhile the JP servers were still up. And
+that&#8217;s pretty much it.</p></div>
+<div class="paragraph"><p>Fast forward a few months and it became pretty clear that
+the private server was going nowhere considering all the drama
+surrounding it-- which is actually not unusual, but it was
+more entertaining than average and the technical abilities of
+people running the project were obviously lacking so I decided
+to obtain those logged packets and look at things myself. I
+didn&#8217;t want to do a private server yet, I only wanted to take
+a peek to see how things worked, and perhaps organize some
+effort to document the protocol.</p></div>
+<div class="paragraph"><p>There was 10GB of logs. I didn&#8217;t have an easy to use
+language to analyze them, and hex editors wouldn&#8217;t cut it for
+most purposes, so I had to look elsewhere. This was a good
+opportunity to start learning this PHP killer I read about
+before, which also happens to feature syntax for matching
+binaries, called Erlang. To be perfectly honest I wouldn&#8217;t
+have touched the logs if I didn&#8217;t have the added motivation
+to play with and learn a new language.</p></div>
+<div class="paragraph"><p>At the time it was pretty hard to learn Erlang. In my
+experience there was Joe&#8217;s book (which I always recommend
+first as I believe it is the best to learn the Erlang side
+of things; but falls a little short on OTP), and there was
+about 5 chapters of LYSE. There were a couple other books
+I never managed to get into (sorry guys), and there was also
+a few interesting blogs, some of which I can&#8217;t find anymore.
+Finally the #erlang IRC community was there but I was strictly
+lurking at the time.</p></div>
+<div class="paragraph"><p>What a difference compared to 4 years later! (That&#8217;s
+today, by the way!) Now we have more books than I can
+remember, tons of articles covering various aspects of the
+language and platform, many targeting beginners but a good
+number of them also about advanced topics. We even have a
+free online book, LYSE, with more than 30 chapters covering
+pretty much everything. Needless to say I never finished
+reading LYSE as it got written slower than I learnt.</p></div>
+<div class="paragraph"><p>Back to 2010. I wrote a parser for the logs, and
+aggregated those results into one CSV file per packet type
+so I could open them in Gnumeric and aggregate some more,
+but manually this time, and draw conclusions on the packet
+structures. That was pretty easy. Even for a beginner.
+Anyone can go from zero to that level in a day or two.
+Then, having mastered binary pattern matching, I wanted
+to learn some more Erlang, by making this aggregation
+faster. What I had done before worked, but I wasn&#8217;t going
+to wait forever to process everything sequentially. So I
+looked and found a project called <code>plists</code> (still exists,
+but not maintained AFAIK). I downloaded that project and
+replaced my <code>lists:</code> calls to <code>plists:</code>.
+Boom. In just a few minutes all logs were processed, and
+I had learnt something new.</p></div>
+<div class="paragraph"><p>It is particularly interesting to note that the lack of
+a package manager or index never bothered me. Neither before
+nor after learning Erlang. My experience with package
+managers was mostly related to Ubuntu, a little Perl and
+Python, and PHP&#8217;s Pear. Let&#8217;s just stay polite and say it
+was always a terrible experience. So searching on the Web
+didn&#8217;t feel awkward, because even if I used a tool or
+website I would have ended up doing a search or two anyway.
+This is in contrast to the package index feature in
+<a href="https://github.com/ninenines/erlang.mk">Erlang.mk</a>,
+which is meant to simplify specifying dependencies more
+than anything: <code>DEPS = cowboy</code>. It does not
+attempt to solve any other problem, and will only attempt
+to solve one extra problem in the near future, which is
+the discovery of packages. So expect some kind of website
+listing packages soon enough.</p></div>
+<div class="paragraph"><p>I want to use this parenthese to also point out that at
+the time there was a very small number of projects out there,
+at least compared to today. While you sometimes hear people
+complain about lack of certain libraries, it is so much
+better now than it was before! The situation improves very
+quickly, so much that it&#8217;s not going to be that big an issue
+soon enough.</p></div>
+<div class="paragraph"><p>Wanting to know more about that game&#8217;s protocol, in the
+year 2010, I ended up starting to write more Erlang code to
+simulate a server and use the server to query the client and
+see what was happening, documenting the packets and so on.
+This eventually lead to a larger project implementing more
+and more until people got their hopes up for a revival of
+the game, all the while the now competing original server
+project died in a stream of drama and technical incompetence.
+Of course, I ended up doing what any good Internet citizen
+would do, I crushed people&#8217;s hopes, but that&#8217;s not important
+to our story. The important part is that before giving up
+on this project, I not only learnt a good deal of Erlang
+and a little deal of OTP (which I did not touch until 6
+months after I started with Erlang; see the paragraph
+about learning material above), but I also had an intriguing
+idea pop into my mind for what would become my greatest
+success yet.</p></div>
+<div class="paragraph"><p>The giving up part was not easy. Having had financial
+difficulties all year 2010 and part of 2009, I resolved
+to travel back to Paris to try and make it. I ended up
+sleeping in offices for 6 months, being hosted by a shady
+person, and hearing my fair share of stories about
+the dark side of business. While there I also worked for
+another company with someone who would end up becoming
+another high profile Erlang developer. The situation
+slowly improved, I started taking part in the #erlang
+IRC discussions, giving up my status of lurker and, a
+few months into 2011, started working on the Apache killer
+project: Cowboy.</p></div>
+<div class="paragraph"><p>This is the part where I probably should get accused of
+racism and other fun things, but I never did. And I think
+that speaks lots about the Erlang community. In all my time
+writing Erlang code, I can count the number of conflicts I
+had with other people on a single hand. This is the nicest
+programming community I have ever seen, by far. And the
+humblest too. The Erlang community feels like Japan. And
+I love Japan. So I love the Erlang community. I can&#8217;t say
+this enough. This is something that stayed true for all
+my time using Erlang, and despite the rise of alternative
+languages that are not Japan the Erlang community has
+remained very Japan.</p></div>
+<div class="paragraph"><p>The first published version of Cowboy was written in
+two weeks. A little before those two weeks, during, and
+a while after, pretty much everything I said on the
+Internets was that Cowboy was going to be the greatest
+HTTP server ever, that the other servers were problematic
+(and just to be clear, Yaws was rarely if ever mentioned,
+due to being in a perceived different league of "full
+featured servers" while Cowboy was a "lightweight server"),
+and that Cowboy will be the best replacement to a Mochiweb
+or Misultin application. This, alongside a lot of time
+spent on IRC telling people to use Cowboy when they were
+asking for an HTTP server to use, probably made me sound
+very annoying. But it worked, and Cowboy started getting
+its first users, despite being only a few weeks old. Of
+course, as soon as I got my very first user, I started
+claiming Cowboy had "a lot of users".</p></div>
+<div class="paragraph"><p>Looking back today I would definitely find myself annoying,
+this wasn&#8217;t just an idle comment there. For about a year,
+maybe a little more, all I ever said was that Cowboy was
+the best. This probably made me a little dumber in the
+process (as if I wasn&#8217;t enough! I know). Being French, I
+sometimes would also say things quite abruptly. To stay
+polite, I probably sounded like an asshole. I learnt to
+stop being so French over time thankfully.</p></div>
+<div class="paragraph"><p>I think what was most important to Cowboy at the time,
+was three things. First, it felt fresh. It was new, had new
+ideas, tried to do things differently and followed "new" old
+best practices (the OTP way-- which was simply too obscure
+for most people at the time). Second, it had me spending
+all my time telling people to use it whenever they were
+looking for an HTTP server. Third, it had me helping people
+get started with it and guide them all the steps of the way.
+Mostly because it didn&#8217;t have a very good documentation, but
+still, hand holding does wonders.</p></div>
+<div class="paragraph"><p>To be able to help people every time they had a problem,
+I did not spend all my days reading IRC. Instead I simply
+made sure to be notified when someone said <code>cowboy</code>.
+The same way many people subscribe to alerts when their
+company is mentioned in the news. Nothing fancy.</p></div>
+<div class="paragraph"><p>Time went on, Cowboy grew, or as some like to say,
+completely destroyed the competition, and many people
+eventually moved from Mochiweb and Misultin to Cowboy.
+And then Roberto Ostinelli stopped Misultin development
+and told everyone to move to Cowboy. This is the most
+humble and selfless act I have ever seen in the programming
+sphere, and I only have one thing to say about it: GG.
+Thanks for the fish. He left me with the tasks of improving
+Cowboy examples, documentation and strongly believed that
+the Misultin interface was more user friendly out of all
+the servers. So I added many examples, as many lines of
+documentation as we have of code, and strongly believe
+that Cowboy 2.0 will be the most user friendly interface
+out of all servers. But only time will tell.</p></div>
+<div class="paragraph"><p>With the rise of the project and the rise in the number
+of users, my previous strategy (completely incidental, by
+the way, and definitely not a well thought out plan to
+become popular) stopped working. It was taking me too much
+time. The important aspects slowly drifted. If I wanted to
+support more users, I would have to spend less time with
+each individual user. This was actually a hard problem.
+You basically have to make people understand they can&#8217;t
+just come to you directly when they have a problem, they
+have to follow proper channels. It becomes less personal,
+and might be felt like you don&#8217;t care about them anymore.
+You have to hurt some people&#8217;s feelings at this point. It
+is quite unfortunate, and also quite difficult to do. There
+is some unwritten rule that says early adopters deserve
+more, but in the real world it never works like this. So
+I probably hurt some people&#8217;s feelings at some point. But
+that&#8217;s okay. Because even if you make sure to be as nice
+as possible when you tell people to go through proper
+channels from now on, some people will still get offended.
+There&#8217;s nothing you can do about it.</p></div>
+<div class="paragraph"><p>From that point onward the important points about the
+project was getting the documentation done, making sure
+people knew about the proper channels to get help and
+report issues, etc. Basically making myself less needed.
+This is quite a contrast with the first days, but I believe
+Cowboy made that transition successfully.</p></div>
+<div class="paragraph"><p>Not only did I win time by not having to hold hands with
+everyone all the time (not that I didn&#8217;t like it, but you
+know, the sweat), but I also won time thanks to the increased
+project popularity. Indeed, the more users you have, the more
+annoying guys there are to tell people to use your project
+and that it&#8217;s the best and everything. Which is great. At
+least, it&#8217;s great if you don&#8217;t pay too much attention to it.
+Sometimes people will give an advice that is, in your opinion,
+a bad advice. And that&#8217;s okay. Don&#8217;t intervene every time
+someone gives a bad advice, learn to let it go. People will
+figure it out. You learn by making mistakes, after all. Use
+this extra time to make sure other people don&#8217;t end up
+giving the same bad advice instead. Fix the code or the
+documentation that led to this mistake. Slowly improve the
+project and make sure it doesn&#8217;t happen again.</p></div>
+<div class="paragraph"><p>This is my story. So far, anyway.</p></div>
+</description> + </item> + + <item> + <title>Cowboy 2.0 and query strings</title> + <link>http://ninenines.eu/articles/cowboy2-qs/</link> + <pubDate>Wed, 20 Aug 2014 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/cowboy2-qs/</guid> + <description><div class="paragraph"><p>Now that Cowboy 1.0 is out, I can spend some of my time thinking
+about Cowboy 2.0 that will be released soon after Erlang/OTP 18.0.
+This entry discusses the proposed changes to query string handling
+in Cowboy.</p></div>
+<div class="paragraph"><p>Cowboy 2.0 will respond to user wishes by simplifying the interface
+of the <code>cowboy_req</code> module. Users want two things: less
+juggling with the Req variable, and more maps. Maps is the only
+dynamic key/value data structure in Erlang that we can match directly
+to extract values, allowing users to greatly simplify their code as
+they don&#8217;t need to call functions to do everything anymore.</p></div>
+<div class="paragraph"><p>Query strings are a good candidate for maps. It&#8217;s a list of
+key/values, so it&#8217;s pretty obvious we can win a lot by using maps.
+However query strings have one difference with maps: they can have
+duplicate keys.</p></div>
+<div class="paragraph"><p>How are we expected to handle duplicate keys? There&#8217;s no standard
+behavior. It&#8217;s up to applications. And looking at what is done in
+the wild, there&#8217;s no de facto standard either. While some ignore
+duplicate keys (keeping the first or the last they find), others
+require duplicate keys to end with <code>[]</code> to automatically
+put the values in a list, or even worse, languages like PHP even
+allow you to do things like <code>key[something][other]</code> and
+create a deep structure for it. Finally some allow any key to have
+duplicates and just gives you lists of key/values.</p></div>
+<div class="paragraph"><p>Cowboy so far had functions to retrieve query string values one
+value at a time, and if there were duplicates it would return the
+first it finds. It also has a function returning the entire list
+with all duplicates, allowing you to filter it to get all of them,
+and another function that returns the raw query string.</p></div>
+<div class="paragraph"><p>What are duplicates used for? Not that many things actually.</p></div>
+<div class="paragraph"><p>One use of duplicate keys is with HTML forms. It is common practice
+to give all related checkboxes the same name so you get a list of
+what&#8217;s been checked. When nothing is checked, nothing is sent at all,
+the key is not in the list.</p></div>
+<div class="paragraph"><p>Another use of duplicate keys is when generating forms. A good
+example of that would be a form that allows uploading any number
+of files. When you add a file, client-side code adds another field
+to the form. Repeat up to a certain limit.</p></div>
+<div class="paragraph"><p>And that&#8217;s about it. Of note is that HTML radio elements share
+the same name too, but only one key/value is sent, so they are not
+relevant here.</p></div>
+<div class="paragraph"><p>Normally this would be the part where I tell you how we solve
+this elegantly. But I had doubts. Why? Because there&#8217;s no good
+solutions to solving only this particular problem.</p></div>
+<div class="paragraph"><p>I then stopped thinking about duplicate keys for a minute and
+started to think about the larger problem.</p></div>
+<div class="paragraph"><p>Query strings are input data. They take a particular form,
+and may be sent as part of the URI or as part of the request
+body. We have other kinds of input data. We have headers and
+cookies and the request body in various forms. We also have
+path segments in URIs.</p></div>
+<div class="paragraph"><p>What do you do with input data? Well you use it to do
+something. But there is one thing that you almost always do
+(and if you don&#8217;t, you really should): you validate it and
+you map it into Erlang terms.</p></div>
+<div class="paragraph"><p>Cowboy left the user take care of validation and conversion
+into Erlang terms so far. Rather, it left the user take care
+of it everywhere except one place. Guess where? That&#8217;s right,
+bindings.</p></div>
+<div class="paragraph"><p>If you define routes with bindings then you have the option
+to provide constraints. Constraints can be used to do two things:
+validate the data and convert it in a more appropriate term. For
+example if you use the <code>int</code> constraint, Cowboy will
+make sure the binding is an integer, and will replace the value
+with the integer representation so that you can use it directly.
+In this particular case it not only routes the URI, but also
+validates and converts the bindings directly.</p></div>
+<div class="paragraph"><p>This is very relevant in the case of our duplicate keys,
+because if we have a list with duplicates of a key, chances
+are we want to convert that into a list of Erlang terms, and
+also make sure that all the elements in this list are expected.</p></div>
+<div class="paragraph"><p>The answer to this particular problem is simple. We need a
+function that will parse the query string and apply constraints.
+But this is not all, there is one other problem to be solved.</p></div>
+<div class="paragraph"><p>The other problem is that for the user some keys are mandatory
+and some are optional. Optional keys include the ones that
+correspond to HTML checkboxes: if the key for one or more
+checkbox is missing from the query string, we still want to
+have an empty list in our map so we can easily match. Matching
+maps is great, but not so much when values might be missing,
+so we have to normalize this data a little.</p></div>
+<div class="paragraph"><p>This problem is solved by allowing a default value. If the
+key is missing and a default exists, set it. If no default
+exists, then the key was mandatory and we want to crash.</p></div>
+<div class="paragraph"><p>I therefore make a proposal for changing the query string
+interface to three functions.</p></div>
+<div class="paragraph"><p>The first function already exists, it is <code>cowboy_req:qs(Req)</code>
+and it returns only the query string binary. No more Req returned.</p></div>
+<div class="paragraph"><p>The second function is a renaming of <code>cowboy_req:qs_vals(Req)</code>
+to something more explicit: <code>cowboy_req:parse_qs(Req)</code>.
+The new name implies that a parsing operation is done. It was implicit
+and cached before. It will be explicit and not cached anymore now.
+Again, no more Req returned.</p></div>
+<div class="paragraph"><p>The third function is the one I mentioned above. I think
+the interface <code>cowboy_req:match_qs(Req, Fields)</code> is
+most appropriate. It returns a normalized map that is the same
+regardless of optional fields being provided with the request,
+allowing for easy matching. It crashes if something went wrong.
+Still no Req returned.</p></div>
+<div class="paragraph"><p>I feel that this three function interface provides everything
+one would need to comfortably write applications. You can get
+low level and get the query string directly; you can get a list
+of key/value binaries without any additional processing and do it
+on your own; or you can get a processed map that contains Erlang
+terms ready to be used.</p></div>
+<div class="paragraph"><p>I strongly believe that by democratizing the constraints to
+more than just bindings, but also to query string, cookies and
+other key/values in Cowboy, we can allow the developer to quickly
+and easily go from HTTP request to Erlang function calls. The
+constraints are reusable functions that can serve as guards
+against unwanted data, providing convenience in the process.</p></div>
+<div class="paragraph"><p>Your handlers will not look like an endless series of calls
+to get and convert the input data, they will instead be just
+one call at the beginning followed by the actual application
+logic, thanks to constraints and maps.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">handle</span></span>(<span style="color: #009900">Req</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span>
+ #{<span style="color: #FF6600">name</span><span style="color: #990000">:=</span><span style="color: #009900">Name</span>, <span style="color: #FF6600">email</span><span style="color: #990000">:=</span><span style="color: #009900">Email</span>, <span style="color: #FF6600">choices</span><span style="color: #990000">:=</span><span style="color: #009900">ChoicesList</span>, <span style="color: #FF6600">remember_me</span><span style="color: #990000">:=</span><span style="color: #009900">RememberMe</span>} <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #000000">cowboy_req:match_qs</span></span>(<span style="color: #009900">Req</span>, [
+ <span style="font-weight: bold"><span style="color: #000080">name</span></span>, {<span style="color: #FF6600">email</span>, <span style="color: #FF6600">email</span>},
+ {<span style="color: #FF6600">choices</span>, <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> <span style="font-weight: bold"><span style="color: #000000">check_choices</span></span><span style="color: #990000">/</span><span style="color: #993399">1</span>, []},
+ {<span style="color: #FF6600">remember_me</span>, <span style="color: #FF6600">boolean</span>, <span style="color: #000080">false</span>}]),
+ <span style="font-weight: bold"><span style="color: #000000">save_choices</span></span>(<span style="color: #009900">Name</span>, <span style="color: #009900">Email</span>, <span style="color: #009900">ChoicesList</span>),
+ <span style="font-weight: bold"><span style="color: #0000FF">if</span></span> <span style="color: #009900">RememberMe</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">create_account</span></span>(<span style="color: #009900">Name</span>, <span style="color: #009900">Email</span>); <span style="color: #000080">true</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">ok</span> <span style="font-weight: bold"><span style="color: #0000FF">end</span></span>,
+ {<span style="color: #FF6600">ok</span>, <span style="color: #009900">Req</span>, <span style="color: #009900">State</span>}<span style="color: #990000">.</span>
+
+<span style="font-weight: bold"><span style="color: #000000">check_choices</span></span>(<span style="color: #990000">&lt;&lt;</span><span style="color: #FF0000">"blue"</span><span style="color: #990000">&gt;&gt;</span>) <span style="color: #990000">-&gt;</span> {<span style="color: #000080">true</span>, <span style="color: #FF6600">blue</span>};
+<span style="font-weight: bold"><span style="color: #000000">check_choices</span></span>(<span style="color: #990000">&lt;&lt;</span><span style="color: #FF0000">"red"</span><span style="color: #990000">&gt;&gt;</span>) <span style="color: #990000">-&gt;</span> {<span style="color: #000080">true</span>, <span style="color: #FF6600">red</span>};
+<span style="font-weight: bold"><span style="color: #000000">check_choices</span></span>(<span style="color: #990000">_</span>) <span style="color: #990000">-&gt;</span> <span style="color: #000080">false</span>;</tt></pre></div></div>
+<div class="paragraph"><p>(Don&#8217;t look too closely at the structure yet.)</p></div>
+<div class="paragraph"><p>As you can see in the above snippet, it becomes really easy
+to go from query string to values. You can also use the map
+directly as it is guaranteed to only contain the keys you
+specified, any extra key is not returned.</p></div>
+<div class="paragraph"><p>This would I believe be a huge step up as we can now
+focus on writing applications instead of translating HTTP
+calls. Cowboy can now take care of it.</p></div>
+<div class="paragraph"><p>And to conclude, this also solves our duplicate keys
+dilemma, as they now automatically become a list of binaries,
+and this list is then checked against constraints that
+will fail if they were not expecting a list. And in the
+example above, it even converts the values to atoms for
+easier manipulation.</p></div>
+<div class="paragraph"><p>As usual, feedback is more than welcome, and I apologize
+for the rocky structure of this post as it contains all the
+thoughts that went into this rather than just the conclusion.</p></div>
+</description> + </item> + + <item> + <title>January 2014 status</title> + <link>http://ninenines.eu/articles/january-2014-status/</link> + <pubDate>Tue, 07 Jan 2014 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/january-2014-status/</guid> + <description><div class="paragraph"><p>I will now be regularly writing posts about project status, plans
+and hopes for the future.</p></div>
+<div class="paragraph"><p>Before that though, there&#8217;s one important news to share.</p></div>
+<div class="paragraph"><p>Until a year ago all development was financed through consulting
+and development services. This worked alright but too much time was
+spent doing things that didn&#8217;t benefit the open source projects.
+And that didn&#8217;t make me happy at all. Because I like being happy
+I stopped that for the most part and spent the year figuring things
+out, experimenting and discussing with people about it.</p></div>
+<div class="paragraph"><p>What makes me happy is answering these "what if" questions.
+Ranch and Cowboy are a direct product of that, as they originate
+from the "what if we could have a server running different protocols
+on different ports but all part of the same application?"; Erlang.mk
+is a bit different: "this works great for me, what if it could
+become the standard solution for building Erlang applications?".</p></div>
+<div class="paragraph"><p>When I successfully answer the question, this becomes a project
+that may end up largely benefiting the Erlang community. I love
+Erlang and I love enabling people to build awesome products based
+on my projects. It&#8217;s a lot more rewarding than activities like
+consulting where you only help one company at a time. And it&#8217;s
+also a much better use of my time as this has a bigger impact on
+the community.</p></div>
+<div class="paragraph"><p>The hard part is to figure out how to be able to spend 100%
+of the time on projects that you basically give away for free,
+and still be able to afford living.</p></div>
+<div class="paragraph"><p>The immediate solution was getting work sponsored by the
+<a href="http://www.leofs.org/">LeoFS project</a>. LeoFS is a great
+distributed file storage that I can only recommend to anyone who
+needs to store files or large pieces of data. The sponsorship
+works pretty great, and spurred development of the SPDY code in
+Cowboy amongst other things, plus a couple upcoming projects
+done more recently and getting a final touch before release.</p></div>
+<div class="paragraph"><p>It turns out sponsoring works great. So I&#8217;m thinking of
+expanding on it and hopefully get enough sponsoring for fulltime
+open source development. So I figured out a few things that
+can give incentive to companies willing to sponsor.</p></div>
+<div class="paragraph"><p>Sponsors can <em>request that a particular version of Cowboy
+be maintained indefinitely</em> (as long as they&#8217;re sponsoring).
+This means fixes will be backported. This doesn&#8217;t include
+features although I can take requests depending on feasability.</p></div>
+<div class="paragraph"><p>Sponsors can <em>have a direct, private line of communication</em>,
+useful when they need help debugging or optimizing their product.</p></div>
+<div class="paragraph"><p>Sponsors can <em>get their name associated with one of the
+project</em> and get a good standing in the community thanks
+to this. They would be featured in the README of the project
+which is viewed by hundreds of developers daily.</p></div>
+<div class="paragraph"><p>Sponsors can <em>be listed on this website</em>. I will modify
+the front page when we get a few more sponsors, they will be
+featured below the carousel of projects.</p></div>
+<div class="paragraph"><p>Please <a href="mailto:[email protected]">contact us</a> if
+you are interested in sponsoring, and say how much you are willing
+to sponsor. The goal here is only to have enough money to make a
+living and attend a few conferences. There&#8217;s an upper limit in the
+amount needed per year, so the more sponsors there are the cheaper
+it becomes to everyone.</p></div>
+<div class="paragraph"><p>The upper limit stems from the new legal entity that will replace
+the current Nine Nines. This is mostly to lower the legal costs and
+simplify the administrative stuff and allow me to dedicate all my
+time on what&#8217;s important. From your point of view it&#8217;s business as
+usual.</p></div>
+<div class="paragraph"><p>Now on to project statuses and future works.</p></div>
+<div class="sect1">
+<h2 id="_cowboy">Cowboy</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Cowboy is getting ready for a 1.0 release. Once multipart support
+is in, all that&#8217;s left is finishing the guide, improving tests and
+finishing moving code to the cowlib project. I hope everything will
+be ready around the time R17B is released.</p></div>
+<div class="paragraph"><p>I already dream of some API breaking changes after 1.0, which
+would essentially become 2.0 when they&#8217;re done. An extensive survey
+will be setup after the 1.0 release to get more information on what
+people like and don&#8217;t like about the API.</p></div>
+<div class="paragraph"><p>And of course, when clients start implementing HTTP/2.0 then we
+will too.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_ranch">Ranch</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Ranch is also getting close to 1.0. I am currently writing a
+test suite for upgrades. After that I also would like to write
+a chaos_monkey test suite and add a getting started chapter to the
+guide.</p></div>
+<div class="paragraph"><p>Ranch is pretty solid otherwise, it&#8217;s hard to foresee new
+features at this point.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_erlang_mk">Erlang.mk</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>I didn&#8217;t expect this project to become popular. Glad it did though.</p></div>
+<div class="paragraph"><p>Windows support is planned, but will require GNU Make 4.
+Thankfully, it&#8217;s available at least through cygwin. Make,
+Git and Erlang will be the only required dependencies
+because the rest of the external calls will be converted to
+using Guile, a Scheme included since GNU Make 4. So it is
+Guile that will download the needed files, magically fill
+the list of modules in the <em>.app</em> file and so on, allowing
+us to provide a truly cross-platform solution without
+losing on the performance we benefit from using Make.</p></div>
+<div class="paragraph"><p>Also note that it is possible to check whether Guile
+is available so we will be able to fallback to the current
+code for older systems.</p></div>
+<div class="paragraph"><p>I am also thinking about adding an extra column to the package
+index, indicating the preferred tag or commit number to be used.
+This would allow us to skip the individual <code>dep</code> lines
+entirely if the information in the package index is good enough.
+And committing that file to your project would be the only thing
+needed to lock the dependencies. Of course if a <code>dep</code>
+line is specified this would instead override the file.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_alien_shaman">Alien Shaman</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>This is the two-parts project requested by the LeoFS team.
+This is essentially a "distributed bigwig". I am hoping to
+have a prototype up in a few days.</p></div>
+<div class="paragraph"><p>Alien is the part that allows writing and enabling probes
+in your nodes. Probes send events which may get filtered before
+being forwarded to their destination. The events may be sent
+to a local process, a remote process, over UDP, TCP or SSL.
+Events may also be received by a process called a relay, which
+may be used to group or aggregate data before it is being sent
+over the network, reducing the footprint overall.</p></div>
+<div class="paragraph"><p>Shaman is the UI for it. It will ultimately be able to display
+any event as long as it&#8217;s configured to do so. Events may be logs,
+numeric values displayed on graphs updated in real time, lists of
+items like processes and so on.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_feedback">Feedback</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>That&#8217;s it for today! There will be another status update once
+Shaman is out. But for now I have to focus on it.</p></div>
+<div class="paragraph"><p>As always, please send feedback on the projects, this post,
+the sponsoring idea, anything really! Thanks.</p></div>
+</div>
+</div>
+</description> + </item> + + <item> + <title>Farwest got funded!</title> + <link>http://ninenines.eu/articles/farwest-funded/</link> + <pubDate>Thu, 27 Jun 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/farwest-funded/</guid> + <description><div class="paragraph"><p>This was a triumph! I&#8217;m making a note here: HUGE SUCCESS!!</p></div>
+<iframe frameborder="0" scrolling="no" height="400px" width"236px" seamless="seamless" src="https://api.bountysource.com/user/fundraisers/83/embed"></iframe>
+<div class="paragraph"><p>It&#8217;s hard to overstate my satisfaction. Thanks to everyone who
+made this possible.</p></div>
+<div class="paragraph"><p>If you have backed this fundraiser, and haven&#8217;t provided your
+personal details yet, please do so quickly so that your rewards
+can be sent!</p></div>
+<div class="paragraph"><p>I am hoping that we will be able to make good use of all that
+money. The details of the expenses will be published regularly
+on the <a href="https://github.com/extend/farwest/wiki/2013-Fundraiser">2013 Fundraiser wiki page</a>,
+giving you full disclosure as to how your money is used.</p></div>
+<div class="paragraph"><p>It will take a little time to get things started, we are in
+summer after all! We will however act quickly to make the
+prototype easy enough to use so that the paid UI work can
+begin. This is also when user contributions will be welcome.</p></div>
+<div class="paragraph"><p>You can see the <a href="https://github.com/extend/farwest/wiki/Roadmap">Roadmap</a>
+to get more information on the current plans. This document will
+get updated as time goes on so check again later to see if you
+can help!</p></div>
+<div class="paragraph"><p>Look at me: still talking when there&#8217;s open source to do!</p></div>
+<div class="paragraph"><p>Thanks again for all your support. I really appreciate it.</p></div>
+</description> + </item> + + <item> + <title>Build Erlang releases with Erlang.mk and Relx</title> + <link>http://ninenines.eu/articles/erlang.mk-and-relx/</link> + <pubDate>Tue, 28 May 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/erlang.mk-and-relx/</guid> + <description><div class="paragraph"><p>Building OTP releases has always been a difficult task. Tools like
+Reltool or Rebar have made this simpler, but
+it&#8217;s no panacea. This article will show you an alternative and
+hopefully much simpler solution.</p></div>
+<div class="paragraph"><p>There is two steps to building a release. First you need to build
+the various OTP applications you want to include in the release. Once
+done, you need to create the release itself, by including the Erlang
+runtime system alongside the applications, a boot script to start the
+node and all its applications, and some configuration files.</p></div>
+<div class="paragraph"><p><a href="https://github.com/extend/erlang.mk">Erlang.mk</a> solves
+the first step. It is an include file for GNU Make. Just
+including it in a Makefile is enough to allow building your project,
+fetching and building dependencies, building documentation, performing
+static analysis and more.</p></div>
+<div class="paragraph"><p><a href="https://github.com/erlware/relx">Relx</a> solves the second
+step. It is a release creation tool, wrapped into a single executable
+file. It doesn&#8217;t require a configuration file. And if you do need one,
+it will be a pretty small one.</p></div>
+<div class="paragraph"><p>Let&#8217;s take a look at the smallest Erlang.mk powered
+Makefile. There is only one thing required: defining the project
+name.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #009900">PROJECT =</span> my_project
+
+include erlang.mk</tt></pre></div></div>
+<div class="paragraph"><p>Simply doing this allows you to build your application by typing
+<code>make</code>, running tests using <code>make tests</code>, and
+more. It will even compile your <em>.dtl</em> files found in the
+<em>templates/</em> directory if you are using ErlyDTL!</p></div>
+<div class="paragraph"><p>Let&#8217;s now take a look at a simplified version of the Makefile for
+this website. I only removed a few targets that were off-topic.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #009900">PROJECT =</span> ninenines
+
+<span style="color: #009900">DEPS =</span> cowboy erlydtl
+<span style="color: #009900">dep_cowboy =</span> https<span style="color: #990000">:</span>//github.com/extend/cowboy.git 0.8.5
+<span style="color: #009900">dep_erlydtl =</span> https<span style="color: #990000">:</span>//github.com/evanmiller/erlydtl.git 4d0dc8fb
+
+<span style="font-weight: bold"><span style="color: #000080">.PHONY:</span></span> release clean-release
+
+<span style="color: #990000">release:</span> clean-release all projects
+ relx -o rel<span style="color: #990000">/</span><span style="color: #009900">$(PROJECT)</span>
+
+<span style="color: #990000">clean-release:</span> clean-projects
+ rm -rf rel<span style="color: #990000">/</span><span style="color: #009900">$(PROJECT)</span>
+
+include erlang.mk</tt></pre></div></div>
+<div class="paragraph"><p>You can see here how to define dependencies. First you list all
+the dependency names, then you have one line per dependency, giving
+the repository URL and the commit number, tag or branch you want.</p></div>
+<div class="paragraph"><p>Then you can see two targets defined, with <code>release</code>
+becoming the default target, because it was defined first. You can
+override the default target <code>all</code>, which builds the
+application and its dependencies, this way.</p></div>
+<div class="paragraph"><p>And as you can see, the <code>release</code> target uses
+Relx to build a release into the <em>rel/ninenines/</em>
+directory. Let&#8217;s take a look at the configuration file for this release.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>{<span style="color: #FF6600">release</span>, {<span style="color: #FF6600">ninenines</span>, <span style="color: #FF0000">"1"</span>}, [<span style="color: #FF6600">ninenines</span>]}<span style="color: #990000">.</span>
+
+{<span style="color: #FF6600">extended_start_script</span>, <span style="color: #000080">true</span>}<span style="color: #990000">.</span>
+{<span style="color: #FF6600">sys_config</span>, <span style="color: #FF0000">"rel/sys.config"</span>}<span style="color: #990000">.</span>
+
+{<span style="color: #FF6600">overlay</span>, [
+ {<span style="color: #FF6600">mkdir</span>, <span style="color: #FF0000">"log"</span>},
+ {<span style="color: #FF6600">copy</span>, <span style="color: #FF0000">"rel/vm.args"</span>,
+ <span style="color: #FF0000">"releases/\{\{release_name\}\}-\{\{release_version\}\}/vm.args"</span>}
+]}<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>The first line defines a release named <code>ninenines</code>, which
+has a version number <code>"1"</code> and includes one application, also
+named <code>ninenines</code>, although it doesn&#8217;t have to.</p></div>
+<div class="paragraph"><p>We then use the <code>extended_start_script</code> option to tell
+Relx that we would like to have a start script that allows
+us to not only start the release, but do so with the node in the
+background, or also to allow us to connect to a running node, and so on.
+This start script has the same features as the one tools like
+Rebar generates.</p></div>
+<div class="paragraph"><p>The rest of the file just makes sure our configuration files are
+where we expect them. Relx will automatically take care
+of your <em>sys.config</em> file as long as you tell it where to
+find it. The <em>vm.args</em> file used by the extended start script
+needs to be handled more explicitly by using an overlay however.</p></div>
+<div class="paragraph"><p>How does Relx find what applications to include?
+By looking at the application dependencies in the <em>.app</em>
+file of each OTP application. Make sure you put all dependencies in
+there, <em>including</em> library applications, and Relx
+will find everything for you.</p></div>
+<div class="paragraph"><p>For example, this release includes the following applications.
+Only what&#8217;s strictly required.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>compiler-4.9.1 crypto-2.3 kernel-2.16.1 ranch-0.8.3 syntax_tools-1.6.11
+cowboy-0.8.5 erlydtl-0.7.0 ninenines-0.2.0 stdlib-1.19.1</code></pre>
+</div></div>
+<div class="paragraph"><p>The <em>sys.config</em> file is standard and
+<a href="http://www.erlang.org/doc/man/config.html">well documented</a>.
+The <em>vm.args</em> file is just an optionally multiline file
+containing all the flags to pass to the Erlang VM, for example
+<code>-name [email protected] -heart</code>.</p></div>
+<div class="paragraph"><p>Building OTP releases has always been a difficult task. Until now.</p></div>
+</description> + </item> + + <item> + <title>Xerl: intermediate module</title> + <link>http://ninenines.eu/articles/xerl-0.5-intermediate-module/</link> + <pubDate>Mon, 25 Mar 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/xerl-0.5-intermediate-module/</guid> + <description><div class="paragraph"><p>Today we will start the work on the intermediate module
+that will be used to run the code for the expressions found
+in our file&#8217;s body, replacing our interpreter.</p></div>
+<div class="paragraph"><p>This is what we want to have when all the work is done:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>xerl -&gt; tokens -&gt; AST -&gt; intermediate -&gt; cerl</code></pre>
+</div></div>
+<div class="paragraph"><p>Today we will perform this work only on the atomic integer
+expression however, so we will not build any module at the end.
+We have a few more things to take care of before getting there.
+This does mean that we completely break compilation of modules
+though, so hopefully we can resolve that soon.</p></div>
+<div class="paragraph"><p>This intermediate representation is in the form of a module
+which contains a single function: <code>run/0</code>. This function
+contains all the expressions from our Xerl source file.</p></div>
+<div class="paragraph"><p>In the case of a Xerl source file only containing the integer
+<code>42</code>, we will obtain the following module ready to
+be executed:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000080">-module</span></span>(<span style="color: #FF6600">'$xerl_intermediate'</span>)<span style="color: #990000">.</span>
+<span style="font-weight: bold"><span style="color: #000080">-export</span></span>([<span style="font-weight: bold"><span style="color: #000000">run</span></span><span style="color: #990000">/</span><span style="color: #993399">0</span>])<span style="color: #990000">.</span>
+
+<span style="font-weight: bold"><span style="color: #000000">run</span></span>() <span style="color: #990000">-&gt;</span>
+ <span style="color: #993399">42</span><span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>Running it will of course give us a result of <code>42</code>,
+the same we had when interpreting expressions.</p></div>
+<div class="paragraph"><p>The resulting Core Erlang code looks like this:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">module</span> <span style="color: #FF6600">'$xerl_intermediate'</span> [<span style="color: #FF6600">'run'</span><span style="color: #990000">/</span><span style="color: #993399">0</span>]
+ <span style="color: #FF6600">attributes</span> []
+<span style="color: #FF6600">'run'</span><span style="color: #990000">/</span><span style="color: #993399">0</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> () <span style="color: #990000">-&gt;</span>
+ <span style="color: #993399">42</span>
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span></tt></pre></div></div>
+<div class="paragraph"><p>The nice thing about doing it like this is that other than the
+definition of the intermediate module and its <code>run/0</code>
+function, we can use the same code we are using for generating
+the final Beam file. It may also be faster than interpreting
+if you have complex modules.</p></div>
+<div class="paragraph"><p>Of course this here only works for the simplest cases, as you
+cannot declare a module or a function inside another Erlang function.
+We will need to wrap these into function calls to the Xerl compiler
+that will take care of compiling them, making them available for
+any subsequent expression. We will also need to pass the environment
+to the <code>run</code> function to keep track of all this.</p></div>
+<div class="paragraph"><p>This does mean that we will have different code for compiling
+<code>fun</code> and <code>mod</code> expressions when creating
+the intermediate module. But the many other expressions don&#8217;t need
+any special care.</p></div>
+<div class="paragraph"><p>Right now we&#8217;ve used the <code>'$xerl_intermediate'</code> atom
+for the intermediate module name because we only have one, but we
+will need to have a more random name later on when we&#8217;ll implement
+modules this way.</p></div>
+<div class="paragraph"><p>The attentive mind will know by now that when compiling a Xerl
+file containing one module, we will need to compile two intermediate
+modules: one for the file body, and one for the module&#8217;s body. Worry
+not though, if we only detect <code>mod</code> instructions in the file
+body, we can just skip this phase.</p></div>
+<div class="paragraph"><p>While we&#8217;re at it, we&#8217;ll modify our code generator to handle lists
+of expressions, which didn&#8217;t actually work with integer literals
+before.</p></div>
+<div class="paragraph"><p>We&#8217;re going to use Core Erlang sequences for running the many
+expressions. Sequences work like <code>let</code>, except no value
+is actually bound. Perfect for our case, since we don&#8217;t support
+binding values at this time anyway.</p></div>
+<div class="paragraph"><p>Sequences have an argument and a body, both being Core Erlang
+expressions. The simplest way to have many expressions is to use
+a simple expression for the argument and a sequence for the rest
+of the expressions. When we encounter the last expression in the
+list, we do not create a sequence.</p></div>
+<div class="paragraph"><p>The result is this very simple function:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">comp_body</span></span>([<span style="color: #009900">Expr</span>]) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000000">expr</span></span>(<span style="color: #009900">Expr</span>);
+<span style="font-weight: bold"><span style="color: #000000">comp_body</span></span>([<span style="color: #009900">Expr</span>|<span style="color: #009900">Exprs</span>]) <span style="color: #990000">-&gt;</span>
+ <span style="color: #009900">Arg</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">expr</span></span>(<span style="color: #009900">Expr</span>),
+ <span style="color: #009900">Body</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">comp_body</span></span>(<span style="color: #009900">Exprs</span>),
+ <span style="font-weight: bold"><span style="color: #000000">cerl:c_seq</span></span>(<span style="color: #009900">Arg</span>, <span style="color: #009900">Body</span>)<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>In the case of our example above, a sequence will not be created,
+we only have one expression. If we were to have <code>42, 43, 44</code>
+in our Xerl source file, we would have a result equivalent to the
+following before optimization:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000080">-module</span></span>(<span style="color: #FF6600">'$xerl_intermediate'</span>)<span style="color: #990000">.</span>
+<span style="font-weight: bold"><span style="color: #000080">-export</span></span>([<span style="font-weight: bold"><span style="color: #000000">run</span></span><span style="color: #990000">/</span><span style="color: #993399">0</span>])<span style="color: #990000">.</span>
+
+<span style="font-weight: bold"><span style="color: #000000">run</span></span>() <span style="color: #990000">-&gt;</span>
+ <span style="color: #993399">42</span>,
+ <span style="color: #993399">43</span>,
+ <span style="color: #993399">44</span><span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>And the result is of course <code>44</code>.</p></div>
+<div class="paragraph"><p>The resulting Core Erlang code looks like this:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">module</span> <span style="color: #FF6600">'$xerl_intermediate'</span> [<span style="color: #FF6600">'run'</span><span style="color: #990000">/</span><span style="color: #993399">0</span>]
+ <span style="color: #FF6600">attributes</span> []
+<span style="color: #FF6600">'run'</span><span style="color: #990000">/</span><span style="color: #993399">0</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> () <span style="color: #990000">-&gt;</span>
+ <span style="color: #FF6600">do</span> <span style="color: #993399">42</span>
+ <span style="color: #FF6600">do</span> <span style="color: #993399">43</span>
+ <span style="color: #993399">44</span>
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span></tt></pre></div></div>
+<div class="paragraph"><p>Feels very lisp-y, right? Yep.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+<a href="https://github.com/extend/xerl/blob/0.5/">View the source</a>
+</p>
+</li>
+</ul></div>
+</description> + </item> + + <item> + <title>Xerl: expression separator</title> + <link>http://ninenines.eu/articles/xerl-0.4-expression-separator/</link> + <pubDate>Fri, 01 Mar 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/xerl-0.4-expression-separator/</guid> + <description><div class="paragraph"><p>As promised we are adding an expression separator this time.
+This will be short and easy.</p></div>
+<div class="paragraph"><p>In the tokenizer we only need to add a line recognizing the
+comma as a valid token.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>, <span style="color: #990000">:</span> {<span style="color: #FF6600">token</span>, {<span style="color: #FF6600">','</span>, <span style="color: #009900">TokenLine</span>}}<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>Then we need to change the following lines in the parser:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span>]<span style="color: #990000">.</span>
+<span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #FF6600">exprs</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span> | <span style="color: #FF6600">'$2'</span>]<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>And add a comma between the expressions on the second line:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span>]<span style="color: #990000">.</span>
+<span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #FF6600">','</span> <span style="color: #FF6600">exprs</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span> | <span style="color: #FF6600">'$3'</span>]<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>That takes care of everything except the optional trailing
+comma at the end of our lists of expressions. We just need an
+additional rule to take care of this.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #FF6600">','</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span>]<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>That&#8217;s it.</p></div>
+<div class="paragraph"><p>Wondering why we don&#8217;t have this optional trailing comma in
+Erlang considering how easy it was and the number of people
+complaining about it? Yeah, me too. But that&#8217;s for someone else
+to answer.</p></div>
+<div class="paragraph"><p>Another change I want to talk about is a simple modification
+of the compiler code to use an <code>#env{}</code> record for
+tracking state instead of passing around individual variables.
+This will be required later on when we make modules into proper
+expressions so I thought it was a good idea to anticipate.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+<a href="https://github.com/extend/xerl/blob/0.4/">View the source</a>
+</p>
+</li>
+</ul></div>
+</description> + </item> + + <item> + <title>Erlang Scalability</title> + <link>http://ninenines.eu/articles/erlang-scalability/</link> + <pubDate>Mon, 18 Feb 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/erlang-scalability/</guid> + <description><div class="paragraph"><p>I would like to share some experience and theories on
+Erlang scalability.</p></div>
+<div class="paragraph"><p>This will be in the form of a series of hints, which
+may or may not be accompanied with explanations as to why
+things are this way, or how they improve or reduce the scalability
+of a system. I will try to do my best to avoid giving falsehoods,
+even if that means a few things won&#8217;t be explained.</p></div>
+<div class="sect1">
+<h2 id="_nifs">NIFs</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>NIFs are considered harmful. I don&#8217;t know any single NIF-based
+library that I would recommend. That doesn&#8217;t mean they should
+all be avoided, just that if you&#8217;re going to want your system to
+scale, you probably shouldn&#8217;t use a NIF.</p></div>
+<div class="paragraph"><p>A common case for using NIFs is JSON processing. The problem
+is that JSON is a highly inefficient data structure (similar
+in inefficiency to XML, although perhaps not as bad). If you can
+avoid using JSON, you probably should. MessagePack can replace
+it in many situations.</p></div>
+<div class="paragraph"><p>Long-running NIFs will take over a scheduler and prevent Erlang
+from efficiently handling many processes.</p></div>
+<div class="paragraph"><p>Short-running NIFs will still confuse the scheduler if they
+take more than a few microseconds to run.</p></div>
+<div class="paragraph"><p>Threaded NIFs, or the use of the <code>enif_consume_timeslice</code>
+might help allievate this problem, but they&#8217;re not a silver bullet.</p></div>
+<div class="paragraph"><p>And as you already know, a crashing NIF will take down your VM,
+destroying any claims you may have at being scalable.</p></div>
+<div class="paragraph"><p>Never use a NIF because "C is fast". This is only true in
+single-threaded programs.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_bifs">BIFs</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>BIFs can also be harmful. While they are generally better than
+NIFs, they are not perfect and some of them might have harmful
+effects on the scheduler.</p></div>
+<div class="paragraph"><p>A great example of this is the <code>erlang:decode_packet/3</code>
+BIF, when used for HTTP request or response decoding. Avoiding
+its use in <em>Cowboy</em> allowed us to see a big increase in
+the number of requests production systems were able to handle,
+up to two times the original amount. Incidentally this is something
+that is impossible to detect using synthetic benchmarks.</p></div>
+<div class="paragraph"><p>BIFs that return immediately are perfectly fine though.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_binary_strings">Binary strings</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Binary strings use less memory, which means you spend less time
+allocating memory compared to list-based strings. They are also
+more natural for strings manipulation because they are optimized
+for appending (as opposed to prepending for lists).</p></div>
+<div class="paragraph"><p>If you can process a binary string using a single match context,
+then the code will run incredibly fast. The effects will be much
+increased if the code was compiled using HiPE, even if your Erlang
+system isn&#8217;t compiled natively.</p></div>
+<div class="paragraph"><p>Avoid using <code>binary:split</code> or <code>binary:replace</code>
+if you can avoid it. They have a certain overhead due to supporting
+many options that you probably don&#8217;t need for most operations.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_buffering_and_streaming">Buffering and streaming</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Use binaries. They are great for appending, and it&#8217;s a direct copy
+from what you receive from a stream (usually a socket or a file).</p></div>
+<div class="paragraph"><p>Be careful to not indefinitely receive data, as you might end up
+having a single binary taking up huge amounts of memory.</p></div>
+<div class="paragraph"><p>If you stream from a socket and know how much data you expect,
+then fetch that data in a single <code>recv</code> call.</p></div>
+<div class="paragraph"><p>Similarly, if you can use a single <code>send</code> call, then
+you should do so, to avoid going back and forth unnecessarily between
+your Erlang process and the Erlang port for your socket.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_list_and_binary_comprehensions">List and binary comprehensions</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Prefer list comprehensions over <code>lists:map/2</code>. The
+compiler will be able to optimize your code greatly, for example
+not creating the result if you don&#8217;t need it. As time goes on,
+more optimizations will be added to the compiler and you will
+automatically benefit from them.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_gen_server_is_no_silver_bullet">gen_server is no silver bullet</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>It&#8217;s a bad idea to use <code>gen_server</code> for everything.
+For two reasons.</p></div>
+<div class="paragraph"><p>There is an overhead everytime the <code>gen_server</code> receives
+a call, a cast or a simple message. It can be a problem if your
+<code>gen_server</code> is in a critical code path where speed
+is all that matters. Do not hesitate to create other kinds of
+processes where it makes sense. And depending on the kind of process,
+you might want to consider making them special processes, which
+would essentially behave the same as a <code>gen_server</code>.</p></div>
+<div class="paragraph"><p>A common mistake is to have a unique <code>gen_server</code> to
+handle queries from many processes. This generally becomes the
+biggest bottleneck you&#8217;ll want to fix. You should try to avoid
+relying on a single process, using a pool if you can.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_supervisor_and_monitoring">Supervisor and monitoring</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>A <code>supervisor</code> is also a <code>gen_server</code>,
+so the previous points also apply to them.</p></div>
+<div class="paragraph"><p>Sometimes you&#8217;re in a situation where you have supervised
+processes but also want to monitor them in one or more other
+processes, effectively duplicating the work. The supervisor
+already knows when processes die, why not use this to our
+advantage?</p></div>
+<div class="paragraph"><p>You can create a custom supervisor process that will perform
+both the supervision and handle exit and other events, allowing
+to avoid the combination of supervising and monitoring which
+can prove harmful when many processes die at once, or when you
+have many short lived processes.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_ets_for_lolspeed_tm">ets for LOLSPEED(tm)</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>If you have data queried or modified by many processes, then
+<code>ets</code> public or protected tables will give you the
+performance boost you require. Do not forget to set the
+<code>read_concurrency</code> or <code>write_concurrency</code>
+options though.</p></div>
+<div class="paragraph"><p>You might also be thrilled to know that Erlang R16B will feature
+a big performance improvement for accessing <code>ets</code> tables
+concurrently.</p></div>
+</div>
+</div>
+</description> + </item> + + <item> + <title>Xerl: atomic expressions</title> + <link>http://ninenines.eu/articles/xerl-0.3-atomic-expressions/</link> + <pubDate>Mon, 18 Feb 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/xerl-0.3-atomic-expressions/</guid> + <description><div class="paragraph"><p>We will be adding atomic integer expressions to our language.
+These look as follow in Erlang:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #993399">42</span><span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>And the result of this expression is of course 42.</p></div>
+<div class="paragraph"><p>We will be running this expression at compile time, since we
+don&#8217;t have the means to run code at runtime yet. This will of
+course result in no module being compiled, but that&#8217;s OK, it will
+allow us to discuss a few important things we&#8217;ll have to plan for
+later on.</p></div>
+<div class="paragraph"><p>First, we must of course accept integers in the tokenizer.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>{<span style="color: #009900">D</span>}<span style="color: #990000">+</span> <span style="color: #990000">:</span> {<span style="color: #FF6600">token</span>, {<span style="font-weight: bold"><span style="color: #000080">integer</span></span>, <span style="color: #009900">TokenLine</span>, <span style="font-weight: bold"><span style="color: #000080">list_to_integer</span></span>(<span style="color: #009900">TokenChars</span>)}}<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>We must then accept atomic integer expressions in the parser.
+This is a simple change. The integer token is terminal so we need
+to add it to the list of terminals, and then we only need to add
+it as a possible expression.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">expr</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">integer</span> <span style="color: #990000">:</span> <span style="color: #FF6600">'$1'</span><span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>A file containing only the number 42 (with no terminating dot)
+will give the following result when parsing it. This is incidentally
+the same result as when tokenizing.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>[{<span style="font-weight: bold"><span style="color: #000080">integer</span></span>,<span style="color: #993399">1</span>,<span style="color: #993399">42</span>}]</tt></pre></div></div>
+<div class="paragraph"><p>We must then evaluate it. We&#8217;re going to interpret it for now.
+Since the result of this expression is not stored in a variable,
+we are going to simply print it on the screen and discard it.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">execute</span></span>(<span style="color: #009900">Filename</span>, [{<span style="font-weight: bold"><span style="color: #000080">integer</span></span>, <span style="color: #990000">_</span>, <span style="color: #009900">Int</span>}|<span style="color: #009900">Tail</span>], <span style="color: #009900">Modules</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000000">io:format</span></span>(<span style="color: #FF0000">"integer ~p~n"</span>, [<span style="color: #009900">Int</span>]),
+ <span style="font-weight: bold"><span style="color: #000000">execute</span></span>(<span style="color: #009900">Filename</span>, <span style="color: #009900">Tail</span>, <span style="color: #009900">Modules</span>)<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>You might think by now that what we&#8217;ve done so far this time
+is useless. It brings up many interesting questions though.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+What happens if a file contains two integers?
+</p>
+</li>
+<li>
+<p>
+Can we live without expression separators?
+</p>
+</li>
+<li>
+<p>
+Do we need an interpreter for the compile step?
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>This is what happens when we create a file that contains two
+integers on two separate lines:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>[{<span style="font-weight: bold"><span style="color: #000080">integer</span></span>,<span style="color: #993399">1</span>,<span style="color: #993399">42</span>},{<span style="font-weight: bold"><span style="color: #000080">integer</span></span>,<span style="color: #993399">2</span>,<span style="color: #993399">43</span>}]</tt></pre></div></div>
+<div class="paragraph"><p>And on the same lines:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>[{<span style="font-weight: bold"><span style="color: #000080">integer</span></span>,<span style="color: #993399">1</span>,<span style="color: #993399">42</span>},{<span style="font-weight: bold"><span style="color: #000080">integer</span></span>,<span style="color: #993399">1</span>,<span style="color: #993399">43</span>}]</tt></pre></div></div>
+<div class="paragraph"><p>Does this mean we do not need separators between expressions?
+Not quite. The <code>+</code> and <code>-</code> operators are an
+example of why we can&#8217;t have nice things. They are ambiguous. They
+have two different meanings: make an atomic integer positive or
+negative, or perform an addition or a substraction between two
+integers. Without a separator you won&#8217;t be able to know if the
+following snippet is one or two expressions:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #993399">42</span> <span style="color: #990000">-</span> <span style="color: #993399">12</span></tt></pre></div></div>
+<div class="paragraph"><p>Can we use the line ending as an expression separator then?
+Some languages make whitespace important, often the line
+separator becomes the expression separator. I do not think this
+is the best idea, it can lead to errors. For example the following
+snippet would be two expressions:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #009900">Var</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">some_module:some_function</span></span>() <span style="color: #990000">+</span> <span style="font-weight: bold"><span style="color: #000000">some_module:other_function</span></span>()
+ <span style="color: #990000">+</span> <span style="font-weight: bold"><span style="color: #000000">another_module:another_function</span></span>()</tt></pre></div></div>
+<div class="paragraph"><p>It is not obvious what would happen unless you are a veteran
+of the language, and so we will not go down that road. We will use
+an expression separator just like in Erlang: the comma. We will
+however allow a trailing comma to make copy pasting code easier,
+even if this means some old academics guy will go nuts about it
+later on. This trailing comma will be optional and simply discarded
+by the parser when encountered. We will implement this next.</p></div>
+<div class="paragraph"><p>The question as to how we will handle running expressions
+remains. We have two choices here: we can write an interpreter,
+or we can compile the code and run it. Writing an interpreter
+would require us to do twice the work, and we are lazy, so we will
+not do that.</p></div>
+<div class="paragraph"><p>You might already know that Erlang does not use the same code
+for compiling and for evaluating commands in the shell. The main
+reason for this is that in Erlang everything isn&#8217;t an expression.
+Indeed, the compiler compiles forms which contain expressions,
+but you can&#8217;t have forms in the shell.</p></div>
+<div class="paragraph"><p>How are we going to compile the code that isn&#8217;t part of a module
+then? What do we need to run at compile-time, anyway? The body of
+the file itself, of course. The body of module declarations. That&#8217;s
+about it.</p></div>
+<div class="paragraph"><p>For the file itself, we can simply compile it as a big function
+that will be executed. Then, everytime we encounter a module
+declaration, we will run the compiler on its body, making its body
+essentially a big function that will be executed. The same mechanism
+will be applied when we encounter a module declaration at runtime.</p></div>
+<div class="paragraph"><p>At runtime there&#8217;s nothing else for us to do, the result of this
+operation will load all the compiled modules. At compile time we
+will also want to save them to a file. We&#8217;ll see later how we can
+do that.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+<a href="https://github.com/extend/xerl/blob/0.3/">View the source</a>
+</p>
+</li>
+</ul></div>
+</description> + </item> + + <item> + <title>Xerl: two modules</title> + <link>http://ninenines.eu/articles/xerl-0.2-two-modules/</link> + <pubDate>Sun, 03 Feb 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/xerl-0.2-two-modules/</guid> + <description><div class="paragraph"><p>Everything is an expression.</p></div>
+<div class="paragraph"><p>This sentence carries profound meaning. We will invoke it many
+times over the course of these articles.</p></div>
+<div class="paragraph"><p>If everything is an expression, then the language shouldn&#8217;t have
+any problem with me defining two modules in the same source file.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">mod</span> <span style="color: #FF6600">first_module</span>
+<span style="font-weight: bold"><span style="color: #0000FF">begin</span></span>
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span>
+
+<span style="color: #FF6600">mod</span> <span style="color: #FF6600">second_module</span>
+<span style="font-weight: bold"><span style="color: #0000FF">begin</span></span>
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span></tt></pre></div></div>
+<div class="paragraph"><p>Likewise, it shouldn&#8217;t have any problem with me defining a
+module inside another module.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">mod</span> <span style="color: #FF6600">out_module</span>
+<span style="font-weight: bold"><span style="color: #0000FF">begin</span></span>
+ <span style="color: #FF6600">mod</span> <span style="color: #FF6600">in_module</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">begin</span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF">end</span></span>
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span></tt></pre></div></div>
+<div class="paragraph"><p>Of course, in the context of the Erlang VM, these two snippets
+are equivalent; there is nothing preventing you from calling the
+<code>in_module</code> module from any other module. The <code>mod</code>
+instruction means a module should be created in the Erlang VM,
+with no concept of scope attached.</p></div>
+<div class="paragraph"><p>Still we need to handle both. To do this we will add a step
+between the parser and the code generator that will walk over the
+abstract syntax tree, from here onward shortened as <em>AST</em>,
+and transform the AST by executing it where applicable.</p></div>
+<div class="paragraph"><p>What happens when you execute a <code>mod</code> instruction?
+A module is created. Since we are compiling, that simply means
+the compiler will branch out and create a module.</p></div>
+<div class="paragraph"><p>If everything is an expression, does that mean this will allow
+me to create modules at runtime using the same syntax? Yes, but
+let&#8217;s not get ahead of ourselves yet.</p></div>
+<div class="paragraph"><p>For now we will just iterate over the AST, and will compile
+a module for each <code>mod</code> found. Modules cannot contain
+expressions yet, so there&#8217;s no need to recurse over it at this
+point. This should solve the compilation of our first snippet.</p></div>
+<div class="paragraph"><p>The <code>compile/1</code> function becomes:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #000000">compile</span></span>(<span style="color: #009900">Filename</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000000">io:format</span></span>(<span style="color: #FF0000">"Compiling ~s...~n"</span>, [<span style="color: #009900">Filename</span>]),
+ {<span style="color: #FF6600">ok</span>, <span style="color: #009900">Src</span>} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">file:read_file</span></span>(<span style="color: #009900">Filename</span>),
+ {<span style="color: #FF6600">ok</span>, <span style="color: #009900">Tokens</span>, <span style="color: #990000">_</span>} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">xerl_lexer:string</span></span>(<span style="font-weight: bold"><span style="color: #000080">binary_to_list</span></span>(<span style="color: #009900">Src</span>)),
+ {<span style="color: #FF6600">ok</span>, <span style="color: #009900">Exprs</span>} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">xerl_parser:parse</span></span>(<span style="color: #009900">Tokens</span>),
+ <span style="font-weight: bold"><span style="color: #000000">execute</span></span>(<span style="color: #009900">Filename</span>, <span style="color: #009900">Exprs</span>, [])<span style="color: #990000">.</span>
+
+<span style="font-weight: bold"><span style="color: #000000">execute</span></span>(<span style="color: #990000">_</span>, [], <span style="color: #009900">Modules</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000000">io:format</span></span>(<span style="color: #FF0000">"Done...~n"</span>),
+ {<span style="color: #FF6600">ok</span>, <span style="font-weight: bold"><span style="color: #000000">lists:reverse</span></span>(<span style="color: #009900">Modules</span>)};
+<span style="font-weight: bold"><span style="color: #000000">execute</span></span>(<span style="color: #009900">Filename</span>, [<span style="color: #009900">Expr</span> <span style="color: #990000">=</span> {<span style="color: #FF6600">mod</span>, <span style="color: #990000">_</span>, {<span style="font-weight: bold"><span style="color: #000080">atom</span></span>, <span style="color: #990000">_</span>, <span style="color: #009900">Name</span>}, []}|<span style="color: #009900">Tail</span>], <span style="color: #009900">Modules</span>) <span style="color: #990000">-&gt;</span>
+ {<span style="color: #FF6600">ok</span>, [<span style="color: #009900">Core</span>]} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">xerl_codegen:exprs</span></span>([<span style="color: #009900">Expr</span>]),
+ {<span style="color: #FF6600">ok</span>, [{<span style="color: #009900">Name</span>, []}]} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">core_lint:module</span></span>(<span style="color: #009900">Core</span>),
+ <span style="font-weight: bold"><span style="color: #000000">io:format</span></span>(<span style="color: #FF0000">"~s~n"</span>, [<span style="font-weight: bold"><span style="color: #000000">core_pp:format</span></span>(<span style="color: #009900">Core</span>)]),
+ {<span style="color: #FF6600">ok</span>, <span style="color: #990000">_</span>, <span style="color: #009900">Beam</span>} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">compile:forms</span></span>(<span style="color: #009900">Core</span>,
+ [<span style="font-weight: bold"><span style="color: #000080">binary</span></span>, <span style="color: #FF6600">from_core</span>, <span style="color: #FF6600">return_errors</span>, {<span style="color: #FF6600">source</span>, <span style="color: #009900">Filename</span>}]),
+ {<span style="color: #FF6600">module</span>, <span style="color: #009900">Name</span>} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">code:load_binary</span></span>(<span style="color: #009900">Name</span>, <span style="color: #009900">Filename</span>, <span style="color: #009900">Beam</span>),
+ <span style="font-weight: bold"><span style="color: #000000">execute</span></span>(<span style="color: #009900">Filename</span>, <span style="color: #009900">Tail</span>, [<span style="color: #009900">Name</span>|<span style="color: #009900">Modules</span>])<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>Running this compiler over the first snippet yields the following
+result:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #009900">Compiling</span> <span style="font-weight: bold"><span style="color: #000000">test</span></span><span style="color: #990000">/</span><span style="font-weight: bold"><span style="color: #000000">mod_SUITE_data</span></span><span style="color: #990000">/</span><span style="color: #FF6600">two_modules</span><span style="color: #990000">.</span><span style="color: #FF6600">xerl</span><span style="color: #990000">...</span>
+<span style="color: #FF6600">module</span> <span style="color: #FF6600">'first_module'</span> [<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">0</span>,
+ <span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">1</span>]
+ <span style="color: #FF6600">attributes</span> []
+<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">0</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> () <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000080">call</span></span> <span style="color: #FF6600">'erlang'</span><span style="color: #990000">:</span><span style="color: #FF6600">'get_module_info'</span>
+ (<span style="color: #FF6600">'first_module'</span>)
+<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">1</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> (<span style="color: #009900">Key</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000080">call</span></span> <span style="color: #FF6600">'erlang'</span><span style="color: #990000">:</span><span style="color: #FF6600">'get_module_info'</span>
+ (<span style="color: #FF6600">'first_module'</span>, <span style="color: #009900">Key</span>)
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span>
+<span style="color: #FF6600">module</span> <span style="color: #FF6600">'second_module'</span> [<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">0</span>,
+ <span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">1</span>]
+ <span style="color: #FF6600">attributes</span> []
+<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">0</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> () <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000080">call</span></span> <span style="color: #FF6600">'erlang'</span><span style="color: #990000">:</span><span style="color: #FF6600">'get_module_info'</span>
+ (<span style="color: #FF6600">'second_module'</span>)
+<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">1</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> (<span style="color: #009900">Key</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000080">call</span></span> <span style="color: #FF6600">'erlang'</span><span style="color: #990000">:</span><span style="color: #FF6600">'get_module_info'</span>
+ (<span style="color: #FF6600">'second_module'</span>, <span style="color: #009900">Key</span>)
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span>
+<span style="color: #009900">Done</span><span style="color: #990000">...</span>
+{<span style="color: #FF6600">ok</span>,[<span style="color: #FF6600">first_module</span>,<span style="color: #FF6600">second_module</span>]}</tt></pre></div></div>
+<div class="paragraph"><p>Everything looks fine. And we can check that the two modules have
+been loaded into the VM:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #993399">9</span><span style="color: #990000">&gt;</span> <span style="color: #FF6600">m</span>(<span style="color: #FF6600">first_module</span>)<span style="color: #990000">.</span>
+<span style="color: #009900">Module</span> <span style="color: #FF6600">first_module</span> <span style="color: #FF6600">compiled</span><span style="color: #990000">:</span> <span style="color: #009900">Date</span><span style="color: #990000">:</span> <span style="color: #009900">February</span> <span style="color: #993399">2</span> <span style="color: #993399">2013</span>, <span style="color: #009900">Time</span><span style="color: #990000">:</span> <span style="color: #993399">14.56</span>
+<span style="color: #009900">Compiler</span> <span style="color: #FF6600">options</span><span style="color: #990000">:</span> [<span style="color: #FF6600">from_core</span>]
+<span style="color: #009900">Object</span> <span style="font-weight: bold"><span style="color: #000000">file: test</span></span><span style="color: #990000">/</span><span style="font-weight: bold"><span style="color: #000000">mod_SUITE_data</span></span><span style="color: #990000">/</span><span style="color: #FF6600">two_modules</span><span style="color: #990000">.</span><span style="color: #FF6600">xerl</span>
+<span style="color: #009900">Exports</span><span style="color: #990000">:</span>
+ <span style="font-weight: bold"><span style="color: #000080">module_info</span></span><span style="color: #990000">/</span><span style="color: #993399">0</span>
+ <span style="font-weight: bold"><span style="color: #000080">module_info</span></span><span style="color: #990000">/</span><span style="color: #993399">1</span>
+<span style="color: #FF6600">ok</span>
+<span style="color: #993399">10</span><span style="color: #990000">&gt;</span> <span style="color: #FF6600">m</span>(<span style="color: #FF6600">second_module</span>)<span style="color: #990000">.</span>
+<span style="color: #009900">Module</span> <span style="color: #FF6600">second_module</span> <span style="color: #FF6600">compiled</span><span style="color: #990000">:</span> <span style="color: #009900">Date</span><span style="color: #990000">:</span> <span style="color: #009900">February</span> <span style="color: #993399">2</span> <span style="color: #993399">2013</span>, <span style="color: #009900">Time</span><span style="color: #990000">:</span> <span style="color: #993399">14.56</span>
+<span style="color: #009900">Compiler</span> <span style="color: #FF6600">options</span><span style="color: #990000">:</span> [<span style="color: #FF6600">from_core</span>]
+<span style="color: #009900">Object</span> <span style="font-weight: bold"><span style="color: #000000">file: test</span></span><span style="color: #990000">/</span><span style="font-weight: bold"><span style="color: #000000">mod_SUITE_data</span></span><span style="color: #990000">/</span><span style="color: #FF6600">two_modules</span><span style="color: #990000">.</span><span style="color: #FF6600">xerl</span>
+<span style="color: #009900">Exports</span><span style="color: #990000">:</span>
+ <span style="font-weight: bold"><span style="color: #000080">module_info</span></span><span style="color: #990000">/</span><span style="color: #993399">0</span>
+ <span style="font-weight: bold"><span style="color: #000080">module_info</span></span><span style="color: #990000">/</span><span style="color: #993399">1</span>
+<span style="color: #FF6600">ok</span></tt></pre></div></div>
+<div class="paragraph"><p>So far so good!</p></div>
+<div class="paragraph"><p>What about the second snippet? It brings up many questions. What
+happens once a <code>mod</code> expression has been executed at
+compile time? If it&#8217;s an expression then it has to have a result,
+right? Right. We are still a bit lacking with expressions for now,
+though, so let&#8217;s get back to it after we add more.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+<a href="https://github.com/extend/xerl/blob/0.2/">View the source</a>
+</p>
+</li>
+</ul></div>
+</description> + </item> + + <item> + <title>Xerl: empty modules</title> + <link>http://ninenines.eu/articles/xerl-0.1-empty-modules/</link> + <pubDate>Wed, 30 Jan 2013 00:00:00 +0100</pubDate> + + <guid>http://ninenines.eu/articles/xerl-0.1-empty-modules/</guid> + <description><div class="paragraph"><p>Let&#8217;s build a programming language. I call it Xerl: eXtended ERLang.
+It&#8217;ll be an occasion for us to learn a few things, especially me.</p></div>
+<div class="paragraph"><p>Unlike in Erlang, in this language, everything is an expression.
+This means that modules and functions are expression, and indeed that
+you can have more than one module per file.</p></div>
+<div class="paragraph"><p>We are just starting, so let&#8217;s no go ahead of ourselves here. We&#8217;ll
+begin with writing the code allowing us to compile an empty module.</p></div>
+<div class="paragraph"><p>We will compile to Core Erlang: this is one of the many intermediate
+step your Erlang code compiles to before it becomes BEAM machine code.
+Core Erlang is a very neat language for machine generated code, and we
+will learn many things about it.</p></div>
+<div class="paragraph"><p>Today we will only focus on compiling the following code:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">mod</span> <span style="color: #FF6600">my_module</span>
+<span style="font-weight: bold"><span style="color: #0000FF">begin</span></span>
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span></tt></pre></div></div>
+<div class="paragraph"><p>Compilation will be done in a few steps. First, the source file will
+be transformed in a tree of tokens by the lexer. Then, the parser will
+use that tree of tokens and convert it to the AST, bringing semantical
+meaning to our representation. Finally, the code generator will transform
+this AST to Core Erlang AST, which will then be compiled.</p></div>
+<div class="paragraph"><p>We will use <em>leex</em> for the lexer. This lexer uses .xrl files
+which are then compiled to .erl files that you can then compile to BEAM.
+The file is divided in three parts: definitions, rules and Erlang code.
+Definitions and Erlang code are obvious; rules are what concerns us.</p></div>
+<div class="paragraph"><p>We only need two things: atoms and whitespaces. Atoms are a lowercase
+letter followed by any letter, number, _ or @. Whitespace is either a
+space, an horizontal tab, \r or \n. There exists other kinds of whitespaces
+but we simply do not allow them in the Xerl language.</p></div>
+<div class="paragraph"><p>Rules consist of a regular expression followed by Erlang code. The
+latter must return a token representation or the atom <code>skip_token</code>.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>{<span style="color: #009900">L</span>}{<span style="color: #009900">A</span>}<span style="color: #990000">*</span> <span style="color: #990000">:</span>
+ <span style="color: #009900">Atom</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000080">list_to_atom</span></span>(<span style="color: #009900">TokenChars</span>),
+ {<span style="color: #FF6600">token</span>, <span style="font-weight: bold"><span style="color: #0000FF">case</span></span> <span style="font-weight: bold"><span style="color: #000000">reserved_word</span></span>(<span style="color: #009900">Atom</span>) <span style="font-weight: bold"><span style="color: #0000FF">of</span></span>
+ <span style="color: #000080">true</span> <span style="color: #990000">-&gt;</span> {<span style="color: #009900">Atom</span>, <span style="color: #009900">TokenLine</span>};
+ <span style="color: #000080">false</span> <span style="color: #990000">-&gt;</span> {<span style="font-weight: bold"><span style="color: #000080">atom</span></span>, <span style="color: #009900">TokenLine</span>, <span style="color: #009900">Atom</span>}
+ <span style="font-weight: bold"><span style="color: #0000FF">end</span></span>}<span style="color: #990000">.</span>
+
+{<span style="color: #009900">WS</span>}<span style="color: #990000">+</span> <span style="color: #990000">:</span> <span style="color: #FF6600">skip_token</span><span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p>The first rule matches an atom, which is converted to either a special
+representation for reserved words, or an atom tuple. The
+<code>TokenChars</code> variable represents the match as a string, and
+the <code>TokenLine</code> variable contains the line number.
+<a href="https://github.com/extend/xerl/blob/0.1/src/xerl_lexer.xrl">View the complete file</a>.</p></div>
+<div class="paragraph"><p>We obtain the following result from the lexer:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>[{<span style="color: #FF6600">mod</span>,<span style="color: #993399">1</span>},{<span style="font-weight: bold"><span style="color: #000080">atom</span></span>,<span style="color: #993399">1</span>,<span style="color: #FF6600">my_module</span>},{<span style="color: #FF6600">'begin'</span>,<span style="color: #993399">2</span>},{<span style="color: #FF6600">'end'</span>,<span style="color: #993399">3</span>}]</tt></pre></div></div>
+<div class="paragraph"><p>The second step is to parse this list of tokens to add semantic meaning
+and generate what is called an <em>abstract syntax tree</em>. We will be
+using the <em>yecc</em> parser generator for this. This time it will take
+.yrl files but the process is the same as before. The file is a little
+more complex than for the lexer, we need to define at the very least
+terminals, nonterminals and root symbols, the grammar itself, and
+optionally some Erlang code.</p></div>
+<div class="paragraph"><p>To compile our module, we need a few things. First, everything is an
+expression. We thus need list of expressions and individual expressions.
+We will support a single expression for now, the <code>mod</code>
+expression which defines a module. And that&#8217;s it! We end up with the
+following grammar:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span>]<span style="color: #990000">.</span>
+<span style="color: #FF6600">exprs</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">expr</span> <span style="color: #FF6600">exprs</span> <span style="color: #990000">:</span> [<span style="color: #FF6600">'$1'</span> | <span style="color: #FF6600">'$2'</span>]<span style="color: #990000">.</span>
+
+<span style="color: #FF6600">expr</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">module</span> <span style="color: #990000">:</span> <span style="color: #FF6600">'$1'</span><span style="color: #990000">.</span>
+
+<span style="color: #FF6600">module</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">'mod'</span> <span style="font-weight: bold"><span style="color: #000080">atom</span></span> <span style="color: #FF6600">'begin'</span> <span style="color: #FF6600">'end'</span> <span style="color: #990000">:</span>
+ {<span style="color: #FF6600">'mod'</span>, <span style="font-weight: bold"><span style="color: #000080">?line</span></span>(<span style="color: #FF6600">'$1'</span>), <span style="color: #FF6600">'$2'</span>, []}<span style="color: #990000">.</span></tt></pre></div></div>
+<div class="paragraph"><p><a href="https://github.com/extend/xerl/blob/0.1/src/xerl_parser.yrl">View the complete file</a>.</p></div>
+<div class="paragraph"><p>We obtain the following result from the parser:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt>[{<span style="color: #FF6600">mod</span>,<span style="color: #993399">1</span>,{<span style="font-weight: bold"><span style="color: #000080">atom</span></span>,<span style="color: #993399">1</span>,<span style="color: #FF6600">my_module</span>},[]}]</tt></pre></div></div>
+<div class="paragraph"><p>We obtain a list of a single <code>mod</code> expression. Just like
+we wanted. Last step is generating the Core Erlang code from it.</p></div>
+<div class="paragraph"><p>Code generation generally is comprised of several steps. We will
+discuss these in more details later on. For now, we will focus on the
+minimal needed for successful compilation.</p></div>
+<div class="paragraph"><p>We can use the <code>cerl</code> module to generate Core Erlang AST.
+We will simply be using functions, which allows us to avoid learning
+and keeping up to date with the internal representation.</p></div>
+<div class="paragraph"><p>There&#8217;s one important thing to do when generating Core Erlang AST
+for a module: create the <code>module_info/{0,1}</code> functions.
+Indeed, these are added to Erlang before it becomes Core Erlang, and
+so we need to replicate this ourselves. Do not be concerned however,
+as this only takes a few lines of extra code.</p></div>
+<div class="paragraph"><p>As you can see by
+<a href="https://github.com/extend/xerl/blob/0.1/src/xerl_codegen.erl">looking at the complete file</a>,
+the code generator echoes the grammar we defined in the parser, and
+simply applies the appropriate Core Erlang functions for each expressions.</p></div>
+<div class="paragraph"><p>We obtain the following pretty-printed Core Erlang generated code:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.8
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #FF6600">module</span> <span style="color: #FF6600">'my_module'</span> [<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">0</span>,
+ <span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">1</span>]
+ <span style="color: #FF6600">attributes</span> []
+<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">0</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> () <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000080">call</span></span> <span style="color: #FF6600">'erlang'</span><span style="color: #990000">:</span><span style="color: #FF6600">'get_module_info'</span>
+ (<span style="color: #FF6600">'empty_module'</span>)
+<span style="color: #FF6600">'module_info'</span><span style="color: #990000">/</span><span style="color: #993399">1</span> <span style="color: #990000">=</span>
+ <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> (<span style="color: #009900">Key</span>) <span style="color: #990000">-&gt;</span>
+ <span style="font-weight: bold"><span style="color: #000080">call</span></span> <span style="color: #FF6600">'erlang'</span><span style="color: #990000">:</span><span style="color: #FF6600">'get_module_info'</span>
+ (<span style="color: #FF6600">'empty_module'</span>, <span style="color: #009900">Key</span>)
+<span style="font-weight: bold"><span style="color: #0000FF">end</span></span></tt></pre></div></div>
+<div class="paragraph"><p>For convenience I added all the steps in a <code>xerl:compile/1</code>
+function that you can use against your own .xerl files.</p></div>
+<div class="paragraph"><p>That&#8217;s it for today! We will go into more details over each steps in
+the next few articles.</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+<a href="https://github.com/extend/xerl/blob/0.1/">View the source</a>
+</p>
+</li>
+</ul></div>
+</description> + </item> + + </channel> +</rss>
\ No newline at end of file |