Nine Nines http://ninenines.eu/ Recent content on Nine Nines Hugo -- gohugo.io en-us Mon, 29 Aug 2016 00:00:00 +0100 Mailing list archived http://ninenines.eu/articles/ml-archives/ Mon, 29 Aug 2016 00:00:00 +0100 http://ninenines.eu/articles/ml-archives/ <div class="paragraph"><p>The <a href="http://ninenines.eu/archives/extend">old mailing list archives</a> have been added to the site, mainly for referencing purposes.</p></div> <div class="paragraph"><p>The mailing list has been shut down and all personal information has been deleted.</p></div> <div class="paragraph"><p>If you need help with a project, consider either opening a ticket on that project&#8217;s issues tracker or going through the community channels (erlang-questions, #ninenines or #erlang on Freenode).</p></div> <div class="paragraph"><p>Prefer tickets; often when people have issues it highlights an underlying problem in the project or its documentation.</p></div> <div class="paragraph"><p>Thanks.</p></div> Website update http://ninenines.eu/articles/website-update/ Sat, 02 Apr 2016 00:00:00 +0100 http://ninenines.eu/articles/website-update/ <div class="paragraph"><p>Last week-end I updated the Nine Nines website.</p></div> <div class="paragraph"><p>I switched to <a href="http://gohugo.io/">Hugo</a>. The site is now built from <a href="http://asciidoc.org/">Asciidoc</a> documents. You probably saw me switch to Asciidoc for documentation this past year. This is the natural conclusion to that story. The great thing is that with a little bit of Makefile magic I can just copy the documentation files into Hugo and poof, they appear on the website.</p></div> <div class="paragraph"><p>I am very happy with that new setup. I can now post my thoughts again. Woo! Expect regular posts from now on. I will try to replace my long series of tweets with posts.</p></div> <div class="paragraph"><p>The sections have been rearranged. There used to be a separate training section; now <a href="http://ninenines.eu/services">all my services</a> are described in one page. I have also clarified my areas of expertise. There used to be confusion in the past, so now it should be clearer that I am not a distributed systems expert.</p></div> <div class="paragraph"><p>On that note, if you are looking for my services right now, I&#8217;m not available. I&#8217;ll have to work 7 days a week for a while. Try again in a couple months. More on that in a future post.</p></div> <div class="paragraph"><p>The <a href="http://ninenines.eu/docs">documentation</a> becomes a first class citizen. Bullet and Cowlib don&#8217;t have proper documentation&#8230; yet. I have started working on the Cowlib documentation, and Bullet shouldn&#8217;t take too long. All these projects will be documented when Cowboy gets to 2.0, and will all be supported equally. Note that the Cowboy 1.0 documentation still has the old website templates and links. Don&#8217;t worry about it.</p></div> <div class="paragraph"><p>The mailing lists link has been removed. I did announce a few months back that mailing lists were going to go. They&#8217;re still up right now, but not for long. I am planning to put the archives read-only, link to them from a future post and be done with it. If you have a question, open a ticket on Github. Then I can just decide to leave the ticket open if I want to do improvements based on your feedback.</p></div> <div class="paragraph"><p>I have replaced most of the "we" by "I". I am a one-man company right now. Have been for a while. Doesn&#8217;t make sense to keep a facade. I want to be close to users, not put a barrier between us.</p></div> <div class="paragraph"><p>The RSS changed. The old link doesn&#8217;t work anymore. The new link is at <a href="http://ninenines.eu/index.xml">/index.xml</a>, or <a href="http://ninenines.eu/articles/index.xml">/articles/index.xml</a> if you only care about my posts. I guess that&#8217;s the one most people want.</p></div> <div class="paragraph"><p>I still have some tweaks to do, but it will take a while. My long term plan is to remove Bootstrap, use vanilla CSS and as little JS as possible. The reason for that is that it&#8217;s cheaper than upgrading libraries every few years. Life is too short to spend it upgrading JS libraries.</p></div> The Erlanger Playbook September 2015 Update http://ninenines.eu/articles/erlanger-playbook-september-2015-update/ Wed, 02 Sep 2015 00:00:00 +0100 http://ninenines.eu/articles/erlanger-playbook-september-2015-update/ <div class="paragraph"><p>An update to The Erlanger Playbook is now available!</p></div> <div class="paragraph"><p>The Erlanger Playbook is a book about software development using Erlang. It currently covers all areas from the conception, design, the writing of code, documentation and tests.</p></div> <div class="paragraph"><p>The book is still a work in progress. Future topics will include refactoring, debugging and tracing, benchmarking, releases, community management (for open source projects).</p></div> <div class="paragraph"><p>This update fixes a number of things and adds two chapters: IOlists and Erlang building blocks.</p></div> <div class="paragraph"><p>Learn more about <a href="http://ninenines.eu/articles/erlanger-playbook">The Erlanger Playbook</a>!</p></div> <div class="paragraph"><p>This is a self-published ebook. The base price is 50€. All proceeds will be used to allow me to work on open source full time.</p></div> <div class="paragraph"><p>Thank you for helping me helping you help us all!</p></div> Consulting & Training http://ninenines.eu/services/ Wed, 01 Jul 2015 00:00:00 +0100 http://ninenines.eu/services/ <div class="paragraph"><p>If you are interested by any of these opportunities, <a href="mailto:contact@ninenines.eu">send me an email</a>.</p></div> <div class="sect1"> <h2 id="_consulting">Consulting</h2> <div class="sectionbody"> <div class="paragraph"><p>You can get me, Loïc Hoguin, author of Cowboy, to help you solve a problem or work on a particular project.</p></div> <div class="paragraph"><p>My area of expertise is Erlang; HTTP, Websocket and REST APIs; design and implementation of protocols; and messaging systems.</p></div> <div class="paragraph"><p>I can also be helpful with testing or code reviews.</p></div> <div class="paragraph"><p>I offer both hourly and daily rates:</p></div> <div class="ulist"><ul> <li> <p> 200€ hourly rate (remote) </p> </li> <li> <p> 1000€ daily rate (remote and on-site) </p> </li> </ul></div> <div class="paragraph"><p>For remote consulting, the work can be done by phone, email, IRC, GitHub and/or any other platform for collaborative work.</p></div> <div class="paragraph"><p>For on-site consulting, the travel expenses and accomodations are to be paid by the customer. I will also ask for a higher rate if forced to stay on-site for more than a week.</p></div> <div class="paragraph"><p>Note that my expertise does not cover all areas where Erlang is used. My help will be limited in the areas of distributed databases, or large distributed systems.</p></div> </div> </div> <div class="sect1"> <h2 id="_sponsoring">Sponsoring</h2> <div class="sectionbody"> <div class="paragraph"><p>You can sponsor one of my projects.</p></div> <div class="paragraph"><p>Sponsoring gives you:</p></div> <div class="ulist"><ul> <li> <p> a direct, private line of communication </p> </li> <li> <p> the power to make me maintain older versions of my projects (as long as they are sponsoring) </p> </li> <li> <p> priority when adding features or fixing bugs </p> </li> <li> <p> advertisement space on this website and in the README file of the project of your choice </p> </li> </ul></div> <div class="paragraph"><p>Sponsors may choose to benefit from any of these perks.</p></div> <div class="paragraph"><p>In exchange sponsors must contribute financially. A minimum of 200€ per month is required. Sponsors may give as much as they want. Payment can be monthly or one-time. Invoices are of course provided.</p></div> </div> </div> <div class="sect1"> <h2 id="_erlang_beginner_training">Erlang beginner training</h2> <div class="sectionbody"> <div class="paragraph"><p>I would be happy to introduce more people to Erlang. I have a 1-day Erlang training readily available for consumption. The goal of this training is to teach the basics of Erlang systems and programming. It&#8217;s a kind of "Getting started" for Erlang.</p></div> <div class="paragraph"><p>You can review the <a href="http://ninenines.eu/talks/thinking-in-erlang/thinking-in-erlang.html">training slides</a>.</p></div> <div class="paragraph"><p>This training is meant to be given to a large number of people interested in Erlang, as part of a public event, where anyone interested can come.</p></div> <div class="paragraph"><p>Another important aspect of this training is that it is meant to be affordable. We want the most people to learn Erlang as possible.</p></div> <div class="paragraph"><p>If you have room, think you can gather 20+ people and are interested in sponsoring a training session, then we should talk.</p></div> </div> </div> <div class="sect1"> <h2 id="_custom_training">Custom training</h2> <div class="sectionbody"> <div class="paragraph"><p>I can also provide custom training, tailored to your level and your needs. It can take the form of a class, Q&amp;A or a code review/writing session. I need to know your expectations to prepare an appropriate training.</p></div> <div class="paragraph"><p>Custom training rates are the same as consulting rates and the same restrictions apply.</p></div> </div> </div> Documentation http://ninenines.eu/docs/ Wed, 01 Jul 2015 00:00:00 +0100 http://ninenines.eu/docs/ <div class="sect2"> <h3 id="_contribute">Contribute</h3> <div class="paragraph"><p>Do you have examples, tutorials, videos about one or more of my projects? I would happily include them on this page.</p></div> <div class="paragraph"><p><a href="mailto:contact@ninenines.eu">Send me an email with the details</a>.</p></div> </div> Donate http://ninenines.eu/donate/ Wed, 01 Jul 2015 00:00:00 +0100 http://ninenines.eu/donate/ <div class="sect2"> <h3 id="_like_my_work_donate">Like my work? Donate!</h3> <div class="paragraph"><p>You can donate via Paypal to reward me, Loïc Hoguin, for my work on open source software including Cowboy and Erlang.mk.</p></div> <form action="https://www.paypal.com/cgi-bin/webscr" method="post" style="display:inline"> <input type="hidden" name="cmd" value="_donations"> <input type="hidden" name="business" value="essen@ninenines.eu"> <input type="hidden" name="lc" value="FR"> <input type="hidden" name="item_name" value="Loic Hoguin"> <input type="hidden" name="item_number" value="99s"> <input type="hidden" name="currency_code" value="EUR"> <input type="hidden" name="bn" value="PP-DonationsBF:btn_donate_LG.gif:NonHosted"> <input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!"> <img alt="" border="0" src="https://www.paypalobjects.com/fr_FR/i/scr/pixel.gif" width="1" height="1"> </form> </div> Public talks http://ninenines.eu/talks/ Wed, 01 Jul 2015 00:00:00 +0100 http://ninenines.eu/talks/ <div class="sect2"> <h3 id="_talk_requests">Talk requests</h3> <div class="paragraph"><p>Organizing a conference and in need of a speaker for a talk about Erlang and the Web? Need an introduction to Erlang/OTP for your company? Looking for a cool subject for a user group meeting?</p></div> <div class="paragraph"><p><a href="mailto:contact@ninenines.eu">Send me an email with the details</a>.</p></div> </div> Slogan http://ninenines.eu/slogan/ Wed, 01 Jul 2015 00:00:00 +0100 http://ninenines.eu/slogan/ <div class="paragraph"><p>The Erlanger Playbook is now available!<br /> <a href="http://ninenines.eu/articles/erlanger-playbook">Buy now</a> — <a href="http://ninenines.eu/services">Become a Cowboy project sponsor</a></p></div> The Erlanger Playbook http://ninenines.eu/articles/erlanger-playbook/ Thu, 18 Jun 2015 00:00:00 +0100 http://ninenines.eu/articles/erlanger-playbook/ <div class="paragraph"><p>I am proud to announce the pre-release of The Erlanger Playbook.</p></div> <div class="paragraph"><p>The Erlanger Playbook is a book about software development using Erlang. It currently covers all areas from the conception, design, the writing of code, documentation and tests.</p></div> <div class="paragraph"><p>The book is still a work in progress. Future topics will include refactoring, debugging and tracing, benchmarking, releases, community management (for open source projects).</p></div> <div class="paragraph"><p>The following sections are currently available:</p></div> <div class="ulist"><ul> <li> <p> About this book; Future additions </p> </li> <li> <p> <em>Workflow:</em> Think; Write; Stay productive </p> </li> <li> <p> <em>Documentation:</em> On documentation; Tutorials; User guide; Manual </p> </li> <li> <p> <em>Code:</em> Starting a project; Version control; Project structure; Code style; Best practices; Special processes </p> </li> <li> <p> <em>Tests:</em> On testing; Success typing analysis; Manual testing; Unit testing; Functional testing </p> </li> </ul></div> <div class="paragraph"><p>Read a preview: <a href="http://ninenines.eu/res/erlanger-preview.pdf">Special processes</a></p></div> <div class="paragraph"><p>The book is currently just shy of 100 pages. The final version of the book is planned to be between 200 and 250 pages. A print version of the book will be considered once the final version gets released. The printed book is <strong>not</strong> included in the price.</p></div> <div class="paragraph"><p>This is a self-published book. The base price is 50€. All proceeds will be used to allow me to work on open source full time.</p></div> <form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top"> <input type="hidden" name="cmd" value="_s-xclick"> <input type="hidden" name="hosted_button_id" value="9M44HJCGX3GVN"> <input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!"> <img alt="" border="0" src="https://www.paypalobjects.com/fr_FR/i/scr/pixel.gif" width="1" height="1"> </form> <div class="paragraph"><p>You are more than welcome to pay extra by using this second button. It allows you to set the price you want. Make sure to set it to at least 50€ to receive the book.</p></div> <form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top"> <input type="hidden" name="cmd" value="_s-xclick"> <input type="hidden" name="hosted_button_id" value="BBW9TR9LBK8C2"> <input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!"> <img alt="" border="0" src="https://www.paypalobjects.com/fr_FR/i/scr/pixel.gif" width="1" height="1"> </form> <div class="paragraph"><p>Make sure to provide a valid email address.</p></div> <div class="paragraph"><p>There will be a <strong>delay</strong> between payment and sending of the book. This process is currently manual.</p></div> <div class="paragraph"><p>As the book is a pre-release, feedback is more than welcome. You can send your comments to erlanger@ this website.</p></div> <div class="paragraph"><p>The plan is to add about 20 pages every month until it is completed. You will receive updates to the book for free as soon as they are available.</p></div> <div class="paragraph"><p>Huge thanks for your interest in buying this book!</p></div> Validating UTF-8 binaries with Erlang http://ninenines.eu/articles/erlang-validate-utf8/ Fri, 06 Mar 2015 00:00:00 +0100 http://ninenines.eu/articles/erlang-validate-utf8/ <div class="paragraph"><p>Yesterday I pushed Websocket permessage-deflate to Cowboy master. I also pushed <a href="https://github.com/ninenines/cowlib/commit/7e4983b70ddf8cedb967e36fba6a600731bdad5d">a change in the way the code validates UTF-8 data</a> (required for text and close frames as per the spec).</p></div> <div class="paragraph"><p>When looking into why the permessage-deflate tests in autobahntestsuite were taking such a long time, I found that autobahn is using an adaptation of the algorithm named &lt;a href="http://bjoern.hoehrmann.de/utf-8/decoder/dfa/"&gt;Flexible and Economical UTF-8 Decoder&lt;/a&gt;. This is the C99 implementation:</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-style: italic"><span style="color: #9A1900">// Copyright (c) 2008-2009 Bjoern Hoehrmann &lt;bjoern@hoehrmann.de&gt;</span></span> <span style="font-style: italic"><span style="color: #9A1900">// See http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.</span></span> <span style="font-weight: bold"><span style="color: #000080">#define</span></span> UTF8_ACCEPT <span style="color: #993399">0</span> <span style="font-weight: bold"><span style="color: #000080">#define</span></span> UTF8_REJECT <span style="color: #993399">1</span> <span style="font-weight: bold"><span style="color: #0000FF">static</span></span> <span style="font-weight: bold"><span style="color: #0000FF">const</span></span> <span style="color: #008080">uint8_t</span> utf8d<span style="color: #990000">[]</span> <span style="color: #990000">=</span> <span style="color: #FF0000">{</span> <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 00..1f</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 20..3f</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 40..5f</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 60..7f</span></span> <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span><span style="color: #993399">9</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// 80..9f</span></span> <span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span><span style="color: #993399">7</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// a0..bf</span></span> <span style="color: #993399">8</span><span style="color: #990000">,</span><span style="color: #993399">8</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// c0..df</span></span> <span style="color: #993399">0xa</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x4</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// e0..ef</span></span> <span style="color: #993399">0xb</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x5</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// f0..ff</span></span> <span style="color: #993399">0x0</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x2</span><span style="color: #990000">,</span><span style="color: #993399">0x3</span><span style="color: #990000">,</span><span style="color: #993399">0x5</span><span style="color: #990000">,</span><span style="color: #993399">0x8</span><span style="color: #990000">,</span><span style="color: #993399">0x7</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x4</span><span style="color: #990000">,</span><span style="color: #993399">0x6</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span><span style="color: #993399">0x1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s0..s0</span></span> <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">0</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s1..s2</span></span> <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s3..s4</span></span> <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s5..s6</span></span> <span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">3</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="font-style: italic"><span style="color: #9A1900">// s7..s8</span></span> <span style="color: #FF0000">}</span><span style="color: #990000">;</span> uint32_t inline <span style="font-weight: bold"><span style="color: #000000">decode</span></span><span style="color: #990000">(</span>uint32_t<span style="color: #990000">*</span> state<span style="color: #990000">,</span> uint32_t<span style="color: #990000">*</span> codep<span style="color: #990000">,</span> <span style="color: #008080">uint32_t</span> byte<span style="color: #990000">)</span> <span style="color: #FF0000">{</span> <span style="color: #008080">uint32_t</span> type <span style="color: #990000">=</span> utf8d<span style="color: #990000">[</span>byte<span style="color: #990000">];</span> <span style="color: #990000">*</span>codep <span style="color: #990000">=</span> <span style="color: #990000">(*</span>state <span style="color: #990000">!=</span> UTF8_ACCEPT<span style="color: #990000">)</span> <span style="color: #990000">?</span> <span style="color: #990000">(</span>byte <span style="color: #990000">&amp;</span> <span style="color: #993399">0x3fu</span><span style="color: #990000">)</span> <span style="color: #990000">|</span> <span style="color: #990000">(*</span>codep <span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">6</span><span style="color: #990000">)</span> <span style="color: #990000">:</span> <span style="color: #990000">(</span><span style="color: #993399">0xff</span> <span style="color: #990000">&gt;&gt;</span> type<span style="color: #990000">)</span> <span style="color: #990000">&amp;</span> <span style="color: #990000">(</span>byte<span style="color: #990000">);</span> <span style="color: #990000">*</span>state <span style="color: #990000">=</span> utf8d<span style="color: #990000">[</span><span style="color: #993399">256</span> <span style="color: #990000">+</span> <span style="color: #990000">*</span>state<span style="color: #990000">*</span><span style="color: #993399">16</span> <span style="color: #990000">+</span> type<span style="color: #990000">];</span> <span style="font-weight: bold"><span style="color: #0000FF">return</span></span> <span style="color: #990000">*</span>state<span style="color: #990000">;</span> <span style="color: #FF0000">}</span></tt></pre></div></div> <div class="paragraph"><p>And this is the Erlang implementation I came up with:</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-style: italic"><span style="color: #9A1900">%% This function returns 0 on success, 1 on error, and 2..8 on incomplete data.</span></span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;&gt;&gt;</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span> <span style="color: #009900">State</span>; <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">128</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">8</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">128</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">144</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">4</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">160</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">192</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">194</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">224</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">224</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">4</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">225</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">237</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">237</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">5</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">238</span>; <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">239</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">240</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">6</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">241</span>; <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">242</span>; <span style="color: #009900">C</span> <span style="color: #990000">=:=</span> <span style="color: #993399">243</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">7</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">244</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">8</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">_</span>, <span style="color: #990000">_</span>) <span style="color: #990000">-&gt;</span> <span style="color: #993399">1</span><span style="color: #990000">.</span></tt></pre></div></div> <div class="paragraph"><p>Does it look similar to you? So how did we get there?</p></div> <div class="paragraph"><p>I started with a naive implementation of the original. First, we don&#8217;t need the codepoint calculated and extracted for our validation function. We just want to know the data is valid, so we only need to calculate the next state. Then, the only thing we needed to be careful about was that tuples are 1-based, and that we need to stop processing the binary when we get the state 1 or when the binary is empty.</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;&gt;&gt;</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span> <span style="color: #009900">State</span>; <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">_</span>, <span style="color: #993399">1</span>) <span style="color: #990000">-&gt;</span> <span style="color: #993399">1</span>; <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">257</span> <span style="color: #990000">+</span> <span style="color: #009900">State</span> <span style="color: #990000">*</span> <span style="color: #993399">16</span> <span style="color: #990000">+</span> <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">1</span> <span style="color: #990000">+</span> <span style="color: #009900">C</span>, <span style="font-weight: bold"><span style="color: #000080">?UTF8D</span></span>), <span style="font-weight: bold"><span style="color: #000080">?UTF8D</span></span>))<span style="color: #990000">.</span></tt></pre></div></div> <div class="paragraph"><p>The macro <code>?UTF8D</code> is the tuple equivalent of the C array in the original code.</p></div> <div class="paragraph"><p>Compared to our previous algorithm, this performed about the same. In some situations a little faster, in some a little slower. In other words, not good enough. But because this new algorithm allows us to avoid a binary concatenation this warranted looking further.</p></div> <div class="paragraph"><p>It was time to step into crazy land.</p></div> <div class="paragraph"><p>Erlang is very good at pattern matching, even more so than doing some arithmetic coupled by fetching elements from a tuple. So I decided I was going to write all possible clauses for all combinations of <code>C</code> and <code>State</code>. And by write I mean generate.</p></div> <div class="paragraph"><p>So I opened my Erlang shell, defined the variable <code>D</code> to be the tuple <code>?UTF8D</code> with its 400 elements, and then ran the following expression (after a bit of trial and error):</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="color: #993399">16</span><span style="color: #990000">&gt;</span> <span style="font-weight: bold"><span style="color: #000000">file:write_file</span></span>(<span style="color: #FF0000">"out.txt"</span>, [<span style="font-weight: bold"><span style="color: #000000">io_lib:format</span></span>(<span style="color: #FF0000">"validate_utf8(&lt;&lt; ~p, Rest/bits &gt;&gt;, ~p) -&gt; ~p;~n"</span>, [<span style="color: #009900">C</span>, <span style="color: #009900">S</span>, <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">257</span> <span style="color: #990000">+</span> <span style="color: #009900">S</span> <span style="color: #990000">*</span> <span style="color: #993399">16</span> <span style="color: #990000">+</span> <span style="font-weight: bold"><span style="color: #000080">element</span></span>(<span style="color: #993399">1</span> <span style="color: #990000">+</span> <span style="color: #009900">C</span>, <span style="color: #009900">D</span>), <span style="color: #009900">D</span>)]) || <span style="color: #009900">C</span> <span style="color: #990000">&lt;-</span> <span style="font-weight: bold"><span style="color: #000000">lists:seq</span></span>(<span style="color: #993399">0</span>,<span style="color: #993399">255</span>), <span style="color: #009900">S</span> <span style="color: #990000">&lt;-</span> <span style="font-weight: bold"><span style="color: #000000">lists:seq</span></span>(<span style="color: #993399">0</span>,<span style="color: #993399">8</span>)])<span style="color: #990000">.</span> <span style="color: #FF6600">ok</span></tt></pre></div></div> <div class="paragraph"><p>The result is a 2304 lines long file, containing 2304 clauses. People who pay attention to what I say on Twitter will remember I said something around 3000 clauses, but that was just me not using the right number of states in my estimate.</p></div> <div class="paragraph"><p>There was a little more work to be done on this generated code that I did using regular expressions. We need to recurse when the resulting state is not 1. We also need to stop when the binary is empty, making it the 2305th clause.</p></div> <div class="paragraph"><p>Still, 2305 is a lot. But hey, the code did work, and faster than the previous implementation too! But hey, perhaps I could find a way to reduce its size.</p></div> <div class="paragraph"><p>Removing all the clauses that return 1 and putting a catch-all clause at the end instead reduced the number to about 500, and showed that many clauses were similar:</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">0</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">1</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">2</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">3</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">4</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">5</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">6</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">7</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>);</tt></pre></div></div> <div class="paragraph"><p>But also:</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">157</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #993399">158</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>);</tt></pre></div></div> <div class="paragraph"><p>Patterns, my favorites!</p></div> <div class="paragraph"><p>A little more time was spent to edit the 500 or so clauses into smaller equivalents, testing that performance was not impacted, and comitting the result.</p></div> <div class="paragraph"><p>The patterns above can be found here in the resulting function:</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">0</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">128</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="color: #990000">...</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">2</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">0</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">3</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">5</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">2</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">6</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #990000">&lt;&lt;</span> <span style="color: #009900">C</span>, <span style="color: #009900">Rest</span><span style="color: #990000">/</span><span style="color: #FF6600">bits</span> <span style="color: #990000">&gt;&gt;</span>, <span style="color: #993399">7</span>) <span style="font-weight: bold"><span style="color: #0000FF">when</span></span> <span style="color: #009900">C</span> <span style="color: #990000">&gt;=</span> <span style="color: #993399">144</span>, <span style="color: #009900">C</span> <span style="color: #990000">&lt;</span> <span style="color: #993399">160</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">validate_utf8</span></span>(<span style="color: #009900">Rest</span>, <span style="color: #993399">3</span>); <span style="color: #990000">...</span></tt></pre></div></div> <div class="paragraph"><p>I hope you enjoyed this post.</p></div> On open source http://ninenines.eu/articles/on-open-source/ Fri, 05 Sep 2014 00:00:00 +0100 http://ninenines.eu/articles/on-open-source/ <div class="paragraph"><p>Last week I read a great article <a href="http://videlalvaro.github.io/2014/08/on-contributing-to-opensource.html">on contributing to open source</a> by Alvaro Videla. He makes many great points and I am in agreement with most of it. This made me want to properly explain my point of view with regard to open source and contributions. Unlike most open source evangelism articles I will not talk about ideals or any of that crap, but rather my personal feelings and experience.</p></div> <div class="paragraph"><p>I have been doing open source work for quite some time. My very first open source project was a graphics driver for (the very early version of) the PCSX2 emulator. That was more than ten years ago, and there <a href="http://ngemu.com/threads/gstaris-0-6.30469/">isn&#8217;t much left to look at today</a>. This was followed by a <a href="https://github.com/extend/wee">PHP framework</a> (started long before Zend Framework was even a thing) and a few other small projects. None of them really took off. It&#8217;s alright, that&#8217;s pretty much the fate of most open source projects. You spend a lot of work and sweat and get very little in return from others.</p></div> <div class="paragraph"><p>This sounds harsh but this is the reality of all open source projects. If you are thinking of building a project and releasing it as open source, you should be prepared for that. This is how most of your projects will feel like. Don&#8217;t release a project as open source thinking everyone will pat you on the back and cheer, this won&#8217;t happen. In fact if your project is a too small improvement over existing software, what many people will do is say you have NIH syndrome, regardless of the improvement you bring. So you need not to rely on other people in order to get your enjoyment out of building open source software.</p></div> <div class="paragraph"><p>In my case I get enjoyment from thinking about problems that need solving. Often times the problems are already solved, but nevermind that, I still think about them and sometimes come up with something I feel is better and then write code for it. Writing code is also fun, but not as fun as using my brain to imagine solutions.</p></div> <div class="paragraph"><p>You don&#8217;t need thousands of users to do that. So are users worthless to me then? No, of course not. In fact they are an important component: they bring me problems that need solving. So users are very important to me. But that&#8217;s not the only reason.</p></div> <div class="paragraph"><p>I got lucky that the Cowboy project became popular. And seeing it be this popular, and some of my other projects also do quite well, made me believe I could perhaps work full time on open source. If I can work full time then I can produce better software. What I had one hour to work on before I can now spend a day on, and experiment until I am satisfied. This is very useful because that means I can get it almost right from the beginning, and avoid the million API breaking changes that occured before Cowboy 1.0 was released.</p></div> <div class="paragraph"><p>To be able to work full time on open source however, I need money. This is a largely unspoken topic of open source work. The work is never free. You can download the product for free, but someone has to pay for the work itself. Life is unfortunately not free.</p></div> <div class="paragraph"><p>Large projects and some lucky people have their work sponsored by their employers. Everyone else has to deal with it differently. In my case I was sponsored for a while by the <a href="http://leo-project.net/leofs/">LeoFS</a> project, but that ended. I also had the Farwest fundraiser, which was a success, although the project stalled after that. (Fear not, as Farwest will make a comeback as a conglomerate of Web development projects in the future.) After that I set up the <a href="http://ninenines.eu/support/">sponsoring scheme</a>, which I can proudly say today brings in enough money to cover my food and shelter. Great!</p></div> <div class="paragraph"><p>This is a start, but it&#8217;s of course not enough. Life is a little more than food and shelter, and so I am still looking for sponsors. This is not a very glorious experience, as I am essentially looking for scraps that companies can throw away. Still, if a handful more companies were doing that, not only would I be able to live comfortably, but I would also be able to stop worrying about the future as I could put money on the side for when it gets rough.</p></div> <div class="paragraph"><p>A few companies giving me some scrap money so I could live and work independently is by far the most important thing anyone can do to help my projects, including Cowboy. Yes, they&#8217;re even more important than code contributions, bug reports and feedback. Because this money gives me the time I need to handle the code contributions, bug reports and feedback.</p></div> <div class="paragraph"><p>If Cowboy or another project is a large part of your product or infrastructure, then the best thing you can do is become a sponsor. The second best is opening tickets and/or providing feedback. The third best is providing good code contributions.</p></div> <div class="paragraph"><p>I will not expand on the feedback part. Feedback is very important, and even just a high five or a retweet is already good feedback. It&#8217;s not very complicated.</p></div> <div class="paragraph"><p>I want to expand a little on code contributions however. Not long ago I ran across the term "patch bomb" which means dropping patches and expecting the project maintainers to merge them and maintain them. I receive a lot of patches, and often have to refuse them. Causes for refusal vary. Some patches only benefit the people who submitted them (or a very small number of people). Some patches are not refined enough to be included. Others are out of scope of the project. These are some of the reasons why I refuse patches. Having limited time and resources, I have to focus my efforts on the code used by the larger number of users. I have to prioritize patches from submitters who are reactive and address the issues pointed out. And I have to plainly refuse other patches.</p></div> <div class="paragraph"><p>I believe this wraps up my thoughts on open source. Overall I had a great experience, the Erlang community being nice and understanding of the issues at hand in general. And if the money problem could be solved soon, then I would be one of the luckiest and happiest open source developer on Earth.</p></div> <div class="paragraph"><p>Think about it the next time you see a donation button or a request for funds or sponsoring. You can considerably improve an open source developer&#8217;s life with very little of your company&#8217;s money.</p></div> The story so far http://ninenines.eu/articles/the-story-so-far/ Sat, 23 Aug 2014 00:00:00 +0100 http://ninenines.eu/articles/the-story-so-far/ <div class="paragraph"><p>As I am away from home with little to do (some call this a vacation) I wanted to reflect a little on the story so far, or how I arrived to Erlang and got to where I am now. The raw personal experience. It&#8217;ll be an article that&#8217;s more about social aspect, communities and marketing a project than technical considerations. As a period piece, it will also allow me to reflect on the evolution of Erlang in recent years.</p></div> <div class="paragraph"><p>Once upon a time-- Okay this isn&#8217;t a fairy tale. The story begins with a short chapter in 2010. The year 2010 started with a fairly major event in my life: the US servers for the online game I stopped playing a few months before, but was still involved with through its community, were closing. OMG! Someone found a way to log packets and started working on a private server; meanwhile the JP servers were still up. And that&#8217;s pretty much it.</p></div> <div class="paragraph"><p>Fast forward a few months and it became pretty clear that the private server was going nowhere considering all the drama surrounding it-- which is actually not unusual, but it was more entertaining than average and the technical abilities of people running the project were obviously lacking so I decided to obtain those logged packets and look at things myself. I didn&#8217;t want to do a private server yet, I only wanted to take a peek to see how things worked, and perhaps organize some effort to document the protocol.</p></div> <div class="paragraph"><p>There was 10GB of logs. I didn&#8217;t have an easy to use language to analyze them, and hex editors wouldn&#8217;t cut it for most purposes, so I had to look elsewhere. This was a good opportunity to start learning this PHP killer I read about before, which also happens to feature syntax for matching binaries, called Erlang. To be perfectly honest I wouldn&#8217;t have touched the logs if I didn&#8217;t have the added motivation to play with and learn a new language.</p></div> <div class="paragraph"><p>At the time it was pretty hard to learn Erlang. In my experience there was Joe&#8217;s book (which I always recommend first as I believe it is the best to learn the Erlang side of things; but falls a little short on OTP), and there was about 5 chapters of LYSE. There were a couple other books I never managed to get into (sorry guys), and there was also a few interesting blogs, some of which I can&#8217;t find anymore. Finally the #erlang IRC community was there but I was strictly lurking at the time.</p></div> <div class="paragraph"><p>What a difference compared to 4 years later! (That&#8217;s today, by the way!) Now we have more books than I can remember, tons of articles covering various aspects of the language and platform, many targeting beginners but a good number of them also about advanced topics. We even have a free online book, LYSE, with more than 30 chapters covering pretty much everything. Needless to say I never finished reading LYSE as it got written slower than I learnt.</p></div> <div class="paragraph"><p>Back to 2010. I wrote a parser for the logs, and aggregated those results into one CSV file per packet type so I could open them in Gnumeric and aggregate some more, but manually this time, and draw conclusions on the packet structures. That was pretty easy. Even for a beginner. Anyone can go from zero to that level in a day or two. Then, having mastered binary pattern matching, I wanted to learn some more Erlang, by making this aggregation faster. What I had done before worked, but I wasn&#8217;t going to wait forever to process everything sequentially. So I looked and found a project called <code>plists</code> (still exists, but not maintained AFAIK). I downloaded that project and replaced my <code>lists:</code> calls to <code>plists:</code>. Boom. In just a few minutes all logs were processed, and I had learnt something new.</p></div> <div class="paragraph"><p>It is particularly interesting to note that the lack of a package manager or index never bothered me. Neither before nor after learning Erlang. My experience with package managers was mostly related to Ubuntu, a little Perl and Python, and PHP&#8217;s Pear. Let&#8217;s just stay polite and say it was always a terrible experience. So searching on the Web didn&#8217;t feel awkward, because even if I used a tool or website I would have ended up doing a search or two anyway. This is in contrast to the package index feature in <a href="https://github.com/ninenines/erlang.mk">Erlang.mk</a>, which is meant to simplify specifying dependencies more than anything: <code>DEPS = cowboy</code>. It does not attempt to solve any other problem, and will only attempt to solve one extra problem in the near future, which is the discovery of packages. So expect some kind of website listing packages soon enough.</p></div> <div class="paragraph"><p>I want to use this parenthese to also point out that at the time there was a very small number of projects out there, at least compared to today. While you sometimes hear people complain about lack of certain libraries, it is so much better now than it was before! The situation improves very quickly, so much that it&#8217;s not going to be that big an issue soon enough.</p></div> <div class="paragraph"><p>Wanting to know more about that game&#8217;s protocol, in the year 2010, I ended up starting to write more Erlang code to simulate a server and use the server to query the client and see what was happening, documenting the packets and so on. This eventually lead to a larger project implementing more and more until people got their hopes up for a revival of the game, all the while the now competing original server project died in a stream of drama and technical incompetence. Of course, I ended up doing what any good Internet citizen would do, I crushed people&#8217;s hopes, but that&#8217;s not important to our story. The important part is that before giving up on this project, I not only learnt a good deal of Erlang and a little deal of OTP (which I did not touch until 6 months after I started with Erlang; see the paragraph about learning material above), but I also had an intriguing idea pop into my mind for what would become my greatest success yet.</p></div> <div class="paragraph"><p>The giving up part was not easy. Having had financial difficulties all year 2010 and part of 2009, I resolved to travel back to Paris to try and make it. I ended up sleeping in offices for 6 months, being hosted by a shady person, and hearing my fair share of stories about the dark side of business. While there I also worked for another company with someone who would end up becoming another high profile Erlang developer. The situation slowly improved, I started taking part in the #erlang IRC discussions, giving up my status of lurker and, a few months into 2011, started working on the Apache killer project: Cowboy.</p></div> <div class="paragraph"><p>This is the part where I probably should get accused of racism and other fun things, but I never did. And I think that speaks lots about the Erlang community. In all my time writing Erlang code, I can count the number of conflicts I had with other people on a single hand. This is the nicest programming community I have ever seen, by far. And the humblest too. The Erlang community feels like Japan. And I love Japan. So I love the Erlang community. I can&#8217;t say this enough. This is something that stayed true for all my time using Erlang, and despite the rise of alternative languages that are not Japan the Erlang community has remained very Japan.</p></div> <div class="paragraph"><p>The first published version of Cowboy was written in two weeks. A little before those two weeks, during, and a while after, pretty much everything I said on the Internets was that Cowboy was going to be the greatest HTTP server ever, that the other servers were problematic (and just to be clear, Yaws was rarely if ever mentioned, due to being in a perceived different league of "full featured servers" while Cowboy was a "lightweight server"), and that Cowboy will be the best replacement to a Mochiweb or Misultin application. This, alongside a lot of time spent on IRC telling people to use Cowboy when they were asking for an HTTP server to use, probably made me sound very annoying. But it worked, and Cowboy started getting its first users, despite being only a few weeks old. Of course, as soon as I got my very first user, I started claiming Cowboy had "a lot of users".</p></div> <div class="paragraph"><p>Looking back today I would definitely find myself annoying, this wasn&#8217;t just an idle comment there. For about a year, maybe a little more, all I ever said was that Cowboy was the best. This probably made me a little dumber in the process (as if I wasn&#8217;t enough! I know). Being French, I sometimes would also say things quite abruptly. To stay polite, I probably sounded like an asshole. I learnt to stop being so French over time thankfully.</p></div> <div class="paragraph"><p>I think what was most important to Cowboy at the time, was three things. First, it felt fresh. It was new, had new ideas, tried to do things differently and followed "new" old best practices (the OTP way-- which was simply too obscure for most people at the time). Second, it had me spending all my time telling people to use it whenever they were looking for an HTTP server. Third, it had me helping people get started with it and guide them all the steps of the way. Mostly because it didn&#8217;t have a very good documentation, but still, hand holding does wonders.</p></div> <div class="paragraph"><p>To be able to help people every time they had a problem, I did not spend all my days reading IRC. Instead I simply made sure to be notified when someone said <code>cowboy</code>. The same way many people subscribe to alerts when their company is mentioned in the news. Nothing fancy.</p></div> <div class="paragraph"><p>Time went on, Cowboy grew, or as some like to say, completely destroyed the competition, and many people eventually moved from Mochiweb and Misultin to Cowboy. And then Roberto Ostinelli stopped Misultin development and told everyone to move to Cowboy. This is the most humble and selfless act I have ever seen in the programming sphere, and I only have one thing to say about it: GG. Thanks for the fish. He left me with the tasks of improving Cowboy examples, documentation and strongly believed that the Misultin interface was more user friendly out of all the servers. So I added many examples, as many lines of documentation as we have of code, and strongly believe that Cowboy 2.0 will be the most user friendly interface out of all servers. But only time will tell.</p></div> <div class="paragraph"><p>With the rise of the project and the rise in the number of users, my previous strategy (completely incidental, by the way, and definitely not a well thought out plan to become popular) stopped working. It was taking me too much time. The important aspects slowly drifted. If I wanted to support more users, I would have to spend less time with each individual user. This was actually a hard problem. You basically have to make people understand they can&#8217;t just come to you directly when they have a problem, they have to follow proper channels. It becomes less personal, and might be felt like you don&#8217;t care about them anymore. You have to hurt some people&#8217;s feelings at this point. It is quite unfortunate, and also quite difficult to do. There is some unwritten rule that says early adopters deserve more, but in the real world it never works like this. So I probably hurt some people&#8217;s feelings at some point. But that&#8217;s okay. Because even if you make sure to be as nice as possible when you tell people to go through proper channels from now on, some people will still get offended. There&#8217;s nothing you can do about it.</p></div> <div class="paragraph"><p>From that point onward the important points about the project was getting the documentation done, making sure people knew about the proper channels to get help and report issues, etc. Basically making myself less needed. This is quite a contrast with the first days, but I believe Cowboy made that transition successfully.</p></div> <div class="paragraph"><p>Not only did I win time by not having to hold hands with everyone all the time (not that I didn&#8217;t like it, but you know, the sweat), but I also won time thanks to the increased project popularity. Indeed, the more users you have, the more annoying guys there are to tell people to use your project and that it&#8217;s the best and everything. Which is great. At least, it&#8217;s great if you don&#8217;t pay too much attention to it. Sometimes people will give an advice that is, in your opinion, a bad advice. And that&#8217;s okay. Don&#8217;t intervene every time someone gives a bad advice, learn to let it go. People will figure it out. You learn by making mistakes, after all. Use this extra time to make sure other people don&#8217;t end up giving the same bad advice instead. Fix the code or the documentation that led to this mistake. Slowly improve the project and make sure it doesn&#8217;t happen again.</p></div> <div class="paragraph"><p>This is my story. So far, anyway.</p></div> Cowboy 2.0 and query strings http://ninenines.eu/articles/cowboy2-qs/ Wed, 20 Aug 2014 00:00:00 +0100 http://ninenines.eu/articles/cowboy2-qs/ <div class="paragraph"><p>Now that Cowboy 1.0 is out, I can spend some of my time thinking about Cowboy 2.0 that will be released soon after Erlang/OTP 18.0. This entry discusses the proposed changes to query string handling in Cowboy.</p></div> <div class="paragraph"><p>Cowboy 2.0 will respond to user wishes by simplifying the interface of the <code>cowboy_req</code> module. Users want two things: less juggling with the Req variable, and more maps. Maps is the only dynamic key/value data structure in Erlang that we can match directly to extract values, allowing users to greatly simplify their code as they don&#8217;t need to call functions to do everything anymore.</p></div> <div class="paragraph"><p>Query strings are a good candidate for maps. It&#8217;s a list of key/values, so it&#8217;s pretty obvious we can win a lot by using maps. However query strings have one difference with maps: they can have duplicate keys.</p></div> <div class="paragraph"><p>How are we expected to handle duplicate keys? There&#8217;s no standard behavior. It&#8217;s up to applications. And looking at what is done in the wild, there&#8217;s no de facto standard either. While some ignore duplicate keys (keeping the first or the last they find), others require duplicate keys to end with <code>[]</code> to automatically put the values in a list, or even worse, languages like PHP even allow you to do things like <code>key[something][other]</code> and create a deep structure for it. Finally some allow any key to have duplicates and just gives you lists of key/values.</p></div> <div class="paragraph"><p>Cowboy so far had functions to retrieve query string values one value at a time, and if there were duplicates it would return the first it finds. It also has a function returning the entire list with all duplicates, allowing you to filter it to get all of them, and another function that returns the raw query string.</p></div> <div class="paragraph"><p>What are duplicates used for? Not that many things actually.</p></div> <div class="paragraph"><p>One use of duplicate keys is with HTML forms. It is common practice to give all related checkboxes the same name so you get a list of what&#8217;s been checked. When nothing is checked, nothing is sent at all, the key is not in the list.</p></div> <div class="paragraph"><p>Another use of duplicate keys is when generating forms. A good example of that would be a form that allows uploading any number of files. When you add a file, client-side code adds another field to the form. Repeat up to a certain limit.</p></div> <div class="paragraph"><p>And that&#8217;s about it. Of note is that HTML radio elements share the same name too, but only one key/value is sent, so they are not relevant here.</p></div> <div class="paragraph"><p>Normally this would be the part where I tell you how we solve this elegantly. But I had doubts. Why? Because there&#8217;s no good solutions to solving only this particular problem.</p></div> <div class="paragraph"><p>I then stopped thinking about duplicate keys for a minute and started to think about the larger problem.</p></div> <div class="paragraph"><p>Query strings are input data. They take a particular form, and may be sent as part of the URI or as part of the request body. We have other kinds of input data. We have headers and cookies and the request body in various forms. We also have path segments in URIs.</p></div> <div class="paragraph"><p>What do you do with input data? Well you use it to do something. But there is one thing that you almost always do (and if you don&#8217;t, you really should): you validate it and you map it into Erlang terms.</p></div> <div class="paragraph"><p>Cowboy left the user take care of validation and conversion into Erlang terms so far. Rather, it left the user take care of it everywhere except one place. Guess where? That&#8217;s right, bindings.</p></div> <div class="paragraph"><p>If you define routes with bindings then you have the option to provide constraints. Constraints can be used to do two things: validate the data and convert it in a more appropriate term. For example if you use the <code>int</code> constraint, Cowboy will make sure the binding is an integer, and will replace the value with the integer representation so that you can use it directly. In this particular case it not only routes the URI, but also validates and converts the bindings directly.</p></div> <div class="paragraph"><p>This is very relevant in the case of our duplicate keys, because if we have a list with duplicates of a key, chances are we want to convert that into a list of Erlang terms, and also make sure that all the elements in this list are expected.</p></div> <div class="paragraph"><p>The answer to this particular problem is simple. We need a function that will parse the query string and apply constraints. But this is not all, there is one other problem to be solved.</p></div> <div class="paragraph"><p>The other problem is that for the user some keys are mandatory and some are optional. Optional keys include the ones that correspond to HTML checkboxes: if the key for one or more checkbox is missing from the query string, we still want to have an empty list in our map so we can easily match. Matching maps is great, but not so much when values might be missing, so we have to normalize this data a little.</p></div> <div class="paragraph"><p>This problem is solved by allowing a default value. If the key is missing and a default exists, set it. If no default exists, then the key was mandatory and we want to crash.</p></div> <div class="paragraph"><p>I therefore make a proposal for changing the query string interface to three functions.</p></div> <div class="paragraph"><p>The first function already exists, it is <code>cowboy_req:qs(Req)</code> and it returns only the query string binary. No more Req returned.</p></div> <div class="paragraph"><p>The second function is a renaming of <code>cowboy_req:qs_vals(Req)</code> to something more explicit: <code>cowboy_req:parse_qs(Req)</code>. The new name implies that a parsing operation is done. It was implicit and cached before. It will be explicit and not cached anymore now. Again, no more Req returned.</p></div> <div class="paragraph"><p>The third function is the one I mentioned above. I think the interface <code>cowboy_req:match_qs(Req, Fields)</code> is most appropriate. It returns a normalized map that is the same regardless of optional fields being provided with the request, allowing for easy matching. It crashes if something went wrong. Still no Req returned.</p></div> <div class="paragraph"><p>I feel that this three function interface provides everything one would need to comfortably write applications. You can get low level and get the query string directly; you can get a list of key/value binaries without any additional processing and do it on your own; or you can get a processed map that contains Erlang terms ready to be used.</p></div> <div class="paragraph"><p>I strongly believe that by democratizing the constraints to more than just bindings, but also to query string, cookies and other key/values in Cowboy, we can allow the developer to quickly and easily go from HTTP request to Erlang function calls. The constraints are reusable functions that can serve as guards against unwanted data, providing convenience in the process.</p></div> <div class="paragraph"><p>Your handlers will not look like an endless series of calls to get and convert the input data, they will instead be just one call at the beginning followed by the actual application logic, thanks to constraints and maps.</p></div> <div class="listingblock"> <div class="content"><!-- Generator: GNU source-highlight 3.1.8 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre><tt><span style="font-weight: bold"><span style="color: #000000">handle</span></span>(<span style="color: #009900">Req</span>, <span style="color: #009900">State</span>) <span style="color: #990000">-&gt;</span> #{<span style="color: #FF6600">name</span><span style="color: #990000">:=</span><span style="color: #009900">Name</span>, <span style="color: #FF6600">email</span><span style="color: #990000">:=</span><span style="color: #009900">Email</span>, <span style="color: #FF6600">choices</span><span style="color: #990000">:=</span><span style="color: #009900">ChoicesList</span>, <span style="color: #FF6600">remember_me</span><span style="color: #990000">:=</span><span style="color: #009900">RememberMe</span>} <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">cowboy_req:match_qs</span></span>(<span style="color: #009900">Req</span>, [ <span style="font-weight: bold"><span style="color: #000080">name</span></span>, {<span style="color: #FF6600">email</span>, <span style="color: #FF6600">email</span>}, {<span style="color: #FF6600">choices</span>, <span style="font-weight: bold"><span style="color: #0000FF">fun</span></span> <span style="font-weight: bold"><span style="color: #000000">check_choices</span></span><span style="color: #990000">/</span><span style="color: #993399">1</span>, []}, {<span style="color: #FF6600">remember_me</span>, <span style="color: #FF6600">boolean</span>, <span style="color: #000080">false</span>}]), <span style="font-weight: bold"><span style="color: #000000">save_choices</span></span>(<span style="color: #009900">Name</span>, <span style="color: #009900">Email</span>, <span style="color: #009900">ChoicesList</span>), <span style="font-weight: bold"><span style="color: #0000FF">if</span></span> <span style="color: #009900">RememberMe</span> <span style="color: #990000">-&gt;</span> <span style="font-weight: bold"><span style="color: #000000">create_account</span></span>(<span style="color: #009900">Name</span>, <span style="color: #009900">Email</span>); <span style="color: #000080">true</span> <span style="color: #990000">-&gt;</span> <span style="color: #FF6600">ok</span> <span style="font-weight: bold"><span style="color: #0000FF">end</span></span>, {<span style="color: #FF6600">ok</span>, <span style="color: #009900">Req</span>, <span style="color: #009900">State</span>}<span style="color: #990000">.</span> <span style="font-weight: bold"><span style="color: #000000">check_choices</span></span>(<span style="color: #990000">&lt;&lt;</span><span style="color: #FF0000">"blue"</span><span style="color: #990000">&gt;&gt;</span>) <span style="color: #990000">-&gt;</span> {<span style="color: #000080">true</span>, <span style="color: #FF6600">blue</span>}; <span style="font-weight: bold"><span style="color: #000000">check_choices</span></span>(<span style="color: #990000">&lt;&lt;</span><span style="color: #FF0000">"red"</span><span style="color: #990000">&gt;&gt;</span>) <span style="color: #990000">-&gt;</span> {<span style="color: #000080">true</span>, <span style="color: #FF6600">red</span>}; <span style="font-weight: bold"><span style="color: #000000">check_choices</span></span>(<span style="color: #990000">_</span>) <span style="color: #990000">-&gt;</span> <span style="color: #000080">false</span>;</tt></pre></div></div> <div class="paragraph"><p>(Don&#8217;t look too closely at the structure yet.)</p></div> <div class="paragraph"><p>As you can see in the above snippet, it becomes really easy to go from query string to values. You can also use the map directly as it is guaranteed to only contain the keys you specified, any extra key is not returned.</p></div> <div class="paragraph"><p>This would I believe be a huge step up as we can now focus on writing applications instead of translating HTTP calls. Cowboy can now take care of it.</p></div> <div class="paragraph"><p>And to conclude, this also solves our duplicate keys dilemma, as they now automatically become a list of binaries, and this list is then checked against constraints that will fail if they were not expecting a list. And in the example above, it even converts the values to atoms for easier manipulation.</p></div> <div class="paragraph"><p>As usual, feedback is more than welcome, and I apologize for the rocky structure of this post as it contains all the thoughts that went into this rather than just the conclusion.</p></div> January 2014 status http://ninenines.eu/articles/january-2014-status/ Tue, 07 Jan 2014 00:00:00 +0100 http://ninenines.eu/articles/january-2014-status/ <div class="paragraph"><p>I will now be regularly writing posts about project status, plans and hopes for the future.</p></div> <div class="paragraph"><p>Before that though, there&#8217;s one important news to share.</p></div> <div class="paragraph"><p>Until a year ago all development was financed through consulting and development services. This worked alright but too much time was spent doing things that didn&#8217;t benefit the open source projects. And that didn&#8217;t make me happy at all. Because I like being happy I stopped that for the most part and spent the year figuring things out, experimenting and discussing with people about it.</p></div> <div class="paragraph"><p>What makes me happy is answering these "what if" questions. Ranch and Cowboy are a direct product of that, as they originate from the "what if we could have a server running different protocols on different ports but all part of the same application?"; Erlang.mk is a bit different: "this works great for me, what if it could become the standard solution for building Erlang applications?".</p></div> <div class="paragraph"><p>When I successfully answer the question, this becomes a project that may end up largely benefiting the Erlang community. I love Erlang and I love enabling people to build awesome products based on my projects. It&#8217;s a lot more rewarding than activities like consulting where you only help one company at a time. And it&#8217;s also a much better use of my time as this has a bigger impact on the community.</p></div> <div class="paragraph"><p>The hard part is to figure out how to be able to spend 100% of the time on projects that you basically give away for free, and still be able to afford living.</p></div> <div class="paragraph"><p>The immediate solution was getting work sponsored by the <a href="http://www.leofs.org/">LeoFS project</a>. LeoFS is a great distributed file storage that I can only recommend to anyone who needs to store files or large pieces of data. The sponsorship works pretty great, and spurred development of the SPDY code in Cowboy amongst other things, plus a couple upcoming projects done more recently and getting a final touch before release.</p></div> <div class="paragraph"><p>It turns out sponsoring works great. So I&#8217;m thinking of expanding on it and hopefully get enough sponsoring for fulltime open source development. So I figured out a few things that can give incentive to companies willing to sponsor.</p></div> <div class="paragraph"><p>Sponsors can <em>request that a particular version of Cowboy be maintained indefinitely</em> (as long as they&#8217;re sponsoring). This means fixes will be backported. This doesn&#8217;t include features although I can take requests depending on feasability.</p></div> <div class="paragraph"><p>Sponsors can <em>have a direct, private line of communication</em>, useful when they need help debugging or optimizing their product.</p></div> <div class="paragraph"><p>Sponsors can <em>get their name associated with one of the project</em> and get a good standing in the community thanks to this. They would be featured in the README of the project which is viewed by hundreds of developers daily.</p></div> <div class="paragraph"><p>Sponsors can <em>be listed on this website</em>. I will modify the front page when we get a few more sponsors, they will be featured below the carousel of projects.</p></div> <div class="paragraph"><p>Please <a href="mailto:contact@ninenines.eu">contact us</a> if you are interested in sponsoring, and say how much you are willing to sponsor. The goal here is only to have enough money to make a living and attend a few conferences. There&#8217;s an upper limit in the amount needed per year, so the more sponsors there are the cheaper it becomes to everyone.</p></div> <div class="paragraph"><p>The upper limit stems from the new legal entity that will replace the current Nine Nines. This is mostly to lower the legal costs and simplify the administrative stuff and allow me to dedicate all my time on what&#8217;s important. From your point of view it&#8217;s business as usual.</p></div> <div class="paragraph"><p>Now on to project statuses and future works.</p></div> <div class="sect1"> <h2 id="_cowboy">Cowboy</h2> <div class="sectionbody"> <div class="paragraph"><p>Cowboy is getting ready for a 1.0 release. Once multipart support is in, all that&#8217;s left is finishing the guide, improving tests and finishing moving code to the cowlib project. I hope everything will be ready around the time R17B is released.</p></div> <div class="paragraph"><p>I already dream of some API breaking changes after 1.0, which would essentially become 2.0 when they&#8217;re done. An extensive survey will be setup after the 1.0 release to get more information on what people like and don&#8217;t like about the API.</p></div> <div class="paragraph"><p>And of course, when clients start implementing HTTP/2.0 then we will too.</p></div> </div> </div> <div class="sect1"> <h2 id="_ranch">Ranch</h2> <div class="sectionbody"> <div class="paragraph"><p>Ranch is also getting close to 1.0. I am currently writing a test suite for upgrades. After that I also would like to write a chaos_monkey test suite and add a getting started chapter to the guide.</p></div> <div class="paragraph"><p>Ranch is pretty solid otherwise, it&#8217;s hard to foresee new features at this point.</p></div> </div> </div> <div class="sect1"> <h2 id="_erlang_mk">Erlang.mk</h2> <div class="sectionbody"> <div class="paragraph"><p>I didn&#8217;t expect this project to become popular. Glad it did though.</p></div> <div class="paragraph"><p>Windows support is planned, but will require GNU Make 4. Thankfully, it&#8217;s available at least through cygwin. Make, Git and Erlang will be the only required dependencies because the rest of the external calls will be converted to using Guile, a Scheme included since GNU Make 4. So it is Guile that will download the needed files, magically fill the list of modules in the <em>.app</em> file and so on, allowing us to provide a truly cross-platform solution without losing on the performance we benefit from using Make.</p></div> <div class="paragraph"><p>Also note that it is possible to check whether Guile is available so we will be able to fallback to the current code for older systems.</p></div> <div class="paragraph"><p>I am also thinking about adding an extra column to the package index, indicating the preferred tag or commit number to be used. This would allow us to skip the individual <code>dep</code> lines entirely if the information in the package index is good enough. And committing that file to your project would be the only thing needed to lock the dependencies. Of course if a <code>dep</code> line is specified this would instead override the file.</p></div> </div> </div> <div class="sect1"> <h2 id="_alien_shaman">Alien Shaman</h2> <div class="sectionbody"> <div class="paragraph"><p>This is the two-parts project requested by the LeoFS team. This is essentially a "distributed bigwig". I am hoping to have a prototype up in a few days.</p></div> <div class="paragraph"><p>Alien is the part that allows writing and enabling probes in your nodes. Probes send events which may get filtered before being forwarded to their destination. The events may be sent to a local process, a remote process, over UDP, TCP or SSL. Events may also be received by a process called a relay, which may be used to group or aggregate data before it is being sent over the network, reducing the footprint overall.</p></div> <div class="paragraph"><p>Shaman is the UI for it. It will ultimately be able to display any event as long as it&#8217;s configured to do so. Events may be logs, numeric values displayed on graphs updated in real time, lists of items like processes and so on.</p></div> </div> </div> <div class="sect1"> <h2 id="_feedback">Feedback</h2> <div class="sectionbody"> <div class="paragraph"><p>That&#8217;s it for today! There will be another status update once Shaman is out. But for now I have to focus on it.</p></div> <div class="paragraph"><p>As always, please send feedback on the projects, this post, the sponsoring idea, anything really! Thanks.</p></div> </div> </div> Farwest got funded! http://ninenines.eu/articles/farwest-funded/ Thu, 27 Jun 2013 00:00:00 +0100 http://ninenines.eu/articles/farwest-funded/ <div class="paragraph"><p>This was a triumph! I&#8217;m making a note here: HUGE SUCCESS!!</p></div> <iframe frameborder="0" scrolling="no" height="400px" width"236px" seamless="seamless" src="https://api.bountysource.com/user/fundraisers/83/embed"></iframe> <div class="paragraph"><p>It&#8217;s hard to overstate my satisfaction. Thanks to everyone who made this possible.</p></div> <div class="paragraph"><p>If you have backed this fundraiser, and haven&#8217;t provided your personal details yet, please do so quickly so that your rewards can be sent!</p></div> <div class="paragraph"><p>I am hoping that we will be able to make good use of all that money. The details of the expenses will be published regularly on the <a href="https://github.com/extend/farwest/wiki/2013-Fundraiser">2013 Fundraiser wiki page</a>, giving you full disclosure as to how your money is used.</p></div> <div class="paragraph"><p>It will take a little time to get things started, we are in summer after all! We will however act quickly to make the prototype easy enough to use so that the paid UI work can begin. This is also when user contributions will be welcome.</p></div> <div class="paragraph"><p>You can see the <a href="https://github.com/extend/farwest/wiki/Roadmap">Roadmap</a> to get more information on the current plans. This document will get updated as time goes on so check again later to see if you can help!</p></div> <div class="paragraph"><p>Look at me: still talking when there&#8217;s open source to do!</p></div> <div class="paragraph"><p>Thanks again for all your support. I really appreciate it.</p></div>