Wrap-up the user guide

11 files changed, 292 insertions, 210 deletions

diff --git a/guide/internals.md b/guide/architecture.md
index 0f8adc2..b799a37 100644
--- a/guide/internals.md
+++ b/guide/architecture.md
@@ -1,33 +1,18 @@
-Internals
-=========
-
 Architecture
-------------
+============
 
 Cowboy is a lightweight HTTP server.
 
 It is built on top of Ranch. Please see the Ranch guide for more
-informations.
+information.
+
+One process per connection
+--------------------------
 
 It uses only one process per connection. The process where your
 code runs is the process controlling the socket. Using one process
 instead of two allows for lower memory usage.
 
-It uses binaries. Binaries are more efficient than lists for
-representing strings because they take less memory space. Processing
-performance can vary depending on the operation. Binaries are known
-for generally getting a great boost if the code is compiled natively.
-Please see the HiPE documentation for more details.
-
-Because querying for the current date and time can be expensive,
-Cowboy generates one `Date` header value every second, shares it
-to all other processes, which then simply copy it in the response.
-This allows compliance with HTTP/1.1 with no actual performance loss.
-
-One process for many requests
------------------------------
-
-As previously mentioned, Cowboy only use one process per connection.
 Because there can be more than one request per connection with the
 keepalive feature of HTTP/1.1, that means the same process will be
 used to handle many requests.
@@ -37,32 +22,25 @@ up before terminating the handling of the current request. This may
 include cleaning up the process dictionary, timers, monitoring and
 more.
 
-Lowercase header names
-----------------------
-
-For consistency reasons it has been chosen to convert all header names
-to lowercase binary strings. This prevents the programmer from making
-case mistakes, and is possible because header names are case insensitive.
+Binaries
+--------
 
-This works fine for the large majority of clients. However, some badly
-implemented clients, especially ones found in corporate code or closed
-source products, may not handle header names in a case insensitive manner.
-This means that when Cowboy returns lowercase header names, these clients
-will not find the headers they are looking for.
+It uses binaries. Binaries are more efficient than lists for
+representing strings because they take less memory space. Processing
+performance can vary depending on the operation. Binaries are known
+for generally getting a great boost if the code is compiled natively.
+Please see the HiPE documentation for more details.
 
-A simple way to solve this is to create an `onresponse` hook that will
-format the header names with the expected case.
+Date header
+-----------
 
-``` erlang
-capitalize_hook(Status, Headers, Body, Req) ->
-    Headers2 = [{cowboy_bstr:capitalize_token(N), V}
-        || {N, V} <- Headers],
-    {ok, Req2} = cowboy_req:reply(Status, Headers2, Body, Req),
-    Req2.
-```
+Because querying for the current date and time can be expensive,
+Cowboy generates one `Date` header value every second, shares it
+to all other processes, which then simply copy it in the response.
+This allows compliance with HTTP/1.1 with no actual performance loss.
 
-Improving performance
----------------------
+Max connections
+---------------
 
 By default the maximum number of active connections is set to a
 generally accepted big enough number. This is meant to prevent having
@@ -72,8 +50,3 @@ everything else down, or taking up all the memory.
 Disabling this feature, by setting the `{max_connections, infinity}`
 protocol option, would give you greater performance when you are
 only processing short-lived requests.
-
-Another option is to define platform-specific socket options that
-are known to improve their efficiency.
-
-Please see the Ranch guide for more information.
diff --git a/guide/broken_clients.md b/guide/broken_clients.md
new file mode 100644
index 0000000..97e36ce
--- /dev/null
+++ b/guide/broken_clients.md
@@ -0,0 +1,65 @@
+Dealing with broken clients
+===========================
+
+There exists a very large number of implementations for the
+HTTP protocol. Most widely used clients, like browsers,
+follow the standard quite well, but others may not. In
+particular custom enterprise clients tend to be very badly
+written.
+
+Cowboy tries to follow the standard as much as possible,
+but is not trying to handle very possible special cases.
+Instead Cowboy focuses on the cases reported in the wild,
+on the public Web.
+
+That means clients that ignore the HTTP standard completely
+may fail to understand Cowboy's responses. There are of
+course workarounds. This chapter aims to cover them.
+
+Lowercase headers
+-----------------
+
+Cowboy converts all headers it receives to lowercase, and
+similarly sends back headers all in lowercase. Some broken
+HTTP clients have issues with that.
+
+A simple way to solve this is to create an `onresponse` hook
+that will format the header names with the expected case.
+
+``` erlang
+capitalize_hook(Status, Headers, Body, Req) ->
+    Headers2 = [{cowboy_bstr:capitalize_token(N), V}
+        || {N, V} <- Headers],
+    {ok, Req2} = cowboy_req:reply(Status, Headers2, Body, Req),
+    Req2.
+```
+
+Note that SPDY clients do not have that particular issue
+because the specification explicitly says all headers are
+lowercase, unlike HTTP which allows any case but treats
+them as case insensitive.
+
+Camel-case headers
+------------------
+
+Sometimes it is desirable to keep the actual case used by
+clients, for example when acting as a proxy between two broken
+implementations. There is no easy solution for this other than
+forking the project and editing the `cowboy_protocol` file
+directly.
+
+Chunked transfer-encoding
+-------------------------
+
+Sometimes an HTTP client advertises itself as HTTP/1.1 but
+does not support chunked transfer-encoding. This is invalid
+behavior, as HTTP/1.1 clients are required to support it.
+
+A simple workaround exists in these cases. By changing the
+Req object response state to `waiting_stream`, Cowboy will
+understand that it must use the identity transfer-encoding
+when replying, just like if it was an HTTP/1.0 client.
+
+``` erlang
+Req2 = cowboy_req:set(resp_state, waiting_stream).
+```
diff --git a/guide/getting_started.md b/guide/getting_started.md
index 8c01c7d..d366b17 100644
--- a/guide/getting_started.md
+++ b/guide/getting_started.md
@@ -1,21 +1,20 @@
 Getting started
 ===============
 
-Setting up a working Erlang application is a little more complex than
-for most other languages. The reason is that Erlang is designed to
-build systems and not just simple applications.
-
-An Erlang system is typically comprised of many different nodes,
-each containing many different OTP applications, each containing
-many different modules and running many different processes.
-Nodes may or may not be identical depending on the nature of the
-system.
-
-To get started though, we only need one node that contains your own
-HTTP application, plus the dependencies that it needs, like Cowboy.
-To create our node, we need to build what is called a release. A
-release is a set of files that contain the Erlang VM plus all the
-applications required to run our node.
+Erlang is more than a language, it is also an operating system
+for your applications. Erlang developers rarely write standalone
+modules, they write libraries or applications, and then bundle
+those into what is called a release. A release contains the
+Erlang VM plus all applications required to run the node, so
+it can be pushed to production directly.
+
+This chapter walks you through all the steps of setting up
+Cowboy, writing your first application and generating your first
+release. At the end of this chapter you should know everything
+you need to push your first Cowboy application to production.
+
+Application skeleton
+--------------------
 
 Let's start by creating this application. We will simply call it
 `hello_erlang`. This application will have the following directory
@@ -130,6 +129,9 @@ That's not enough however. Since we are building a Cowboy based
 application, we also need to initialize Cowboy when we start our
 application.
 
+Setting up Cowboy
+-----------------
+
 Cowboy does nothing by default.
 
 Cowboy uses Ranch for handling the connections and provides convenience
@@ -198,6 +200,9 @@ init([]) ->
 
 Finally, we need to write the code for handling incoming requests.
 
+Handling HTTP requests
+----------------------
+
 Cowboy features many kinds of handlers. For this simple example,
 we will just use the plain HTTP handler, which has three callback
 functions: `init/3`, `handle/2` and `terminate/3`. You can find more
@@ -235,6 +240,9 @@ Its usage is documented in the
 
 The code for our application is ready, so let's build a release!
 
+Compiling
+---------
+
 First we need to download `erlang.mk`.
 
 ``` bash
@@ -271,6 +279,9 @@ haven't made any typo when creating the previous files.
 $ make
 ```
 
+Generating the release
+----------------------
+
 That's not all however, as we want to create a working release.
 For that purpose, we need to create a `relx.config` file. When
 this file exists, `erlang.mk` will automatically download `relx`
@@ -299,6 +310,3 @@ $ ./_rel/bin/hello_erlang console
 
 If you then access `http://localhost:8080` using your browser,
 you should receive a nice greet!
-
-You can find many more examples in the `examples/` directory
-of the Cowboy repository.
diff --git a/guide/handlers.md b/guide/handlers.md
deleted file mode 100644
index 5ffe401..0000000
--- a/guide/handlers.md
+++ /dev/null
@@ -1,56 +0,0 @@
-Handlers
-========
-
-Purpose
--------
-
-Handlers are Erlang modules that represent a resource.
-
-Handlers must process the request and send a reply. The nature of the
-reply will vary between handlers.
-
-Different kinds of handlers can be combined in a single module. This
-allows a module to handle both websocket and long-polling code in a
-single place, for example.
-
-Protocol upgrades
------------------
-
-Cowboy features many different handlers: HTTP handlers, loop handlers,
-websocket handlers, REST handlers and static handlers. All of them
-have a common entry point: the `init/3` function.
-
-By default, Cowboy considers your handler to be an HTTP handler.
-
-To switch to a different protocol, like, for example, Websocket,
-you must perform a protocol upgrade. This is done by returning
-a protocol upgrade tuple at the end of `init/3`.
-
-The following snippet upgrades the handler to `my_protocol`.
-
-``` erlang
-init(_Any, _Req, _Opts) ->
-    {upgrade, protocol, my_protocol}.
-```
-
-Cowboy comes with two protocol upgrades: `cowboy_rest` and
-`cowboy_websocket`. Use these values in place of `my_protocol`
-to use them.
-
-Custom protocol upgrades
-------------------------
-
-The `my_protocol` module above will be used for further processing
-of the request. It should use the `cowboy_sub_protocol` behaviour,
-which requires only one callback, `upgrade/4`.
-
-It receives the request object, the middleware environment, and
-the handler this request has been routed to along with its options.
-
-``` erlang
-upgrade(Req, Env, Handler, HandlerOpts) ->
-    %% ...
-```
-
-This callback is expected to behave like any middleware. Please
-see the corresponding chapter for more information.
diff --git a/guide/hooks.md b/guide/hooks.md
index d7e6c72..b2e0c50 100644
--- a/guide/hooks.md
+++ b/guide/hooks.md
@@ -1,8 +1,12 @@
 Hooks
 =====
 
-On request
-----------
+Cowboy provides two hooks. `onrequest` is called once the request
+line and headers have been received. `onresponse` is called just
+before sending the response.
+
+Onrequest
+---------
 
 The `onrequest` hook is called as soon as Cowboy finishes fetching
 the request headers. It occurs before any other processing, including
@@ -39,8 +43,8 @@ debug_hook(Req) ->
 
 Make sure to always return the last request object obtained.
 
-On response
------------
+Onresponse
+----------
 
 The `onresponse` hook is called right before sending the response
 to the socket. It can be used for the purposes of logging responses,
@@ -51,7 +55,7 @@ Note that like the `onrequest` hook, this function MUST NOT crash.
 Cowboy may or may not send a reply if this function crashes. If a reply
 is sent, the hook MUST explicitly provide all headers that are needed.
 
-You can specify the `onresponse` hook when creating the listener also.
+You can specify the `onresponse` hook when creating the listener.
 
 ``` erlang
 cowboy:start_http(my_http_listener, 100,
diff --git a/guide/loop_handlers.md b/guide/loop_handlers.md
index c3d1891..8689453 100644
--- a/guide/loop_handlers.md
+++ b/guide/loop_handlers.md
@@ -1,24 +1,11 @@
 Loop handlers
 =============
 
-Purpose
--------
-
 Loop handlers are a special kind of HTTP handlers used when the
 response can not be sent right away. The handler enters instead
 a receive loop waiting for the right message before it can send
 a response.
 
-They are most useful when performing long-polling operations or
-when using server-sent events.
-
-While the same can be accomplished using plain HTTP handlers,
-it is recommended to use loop handlers because they are well-tested
-and allow using built-in features like hibernation and timeouts.
-
-Usage
------
-
 Loop handlers are used for requests where a response might not
 be immediately available, but where you would like to keep the
 connection open for a while in case the response arrives. The
@@ -29,34 +16,133 @@ partially available and you need to stream the response body
 while the connection is open. The most known example of such
 practice is known as server-sent events.
 
+While the same can be accomplished using plain HTTP handlers,
+it is recommended to use loop handlers because they are well-tested
+and allow using built-in features like hibernation and timeouts.
+
 Loop handlers essentially wait for one or more Erlang messages
 and feed these messages to the `info/3` callback. It also features
 the `init/3` and `terminate/3` callbacks which work the same as
 for plain HTTP handlers.
 
-The following handler waits for a message `{reply, Body}` before
-sending a response. If this message doesn't arrive within 60
-seconds, it gives up and a `204 No Content` will be replied.
-It also hibernates the process to save memory while waiting for
-this message.
+Initialization
+--------------
+
+The `init/3` function must return a `loop` tuple to enable
+loop handler behavior. This tuple may optionally contain
+a timeout value and/or the atom `hibernate` to make the
+process enter hibernation until a message is received.
+
+This snippet enables the loop handler.
+
+``` erlang
+init(_Type, Req, _Opts) ->
+    {loop, Req, undefined_state}.
+```
+
+However it is largely recommended that you set a timeout
+value. The next example sets a timeout value of 30s and
+also makes the process hibernate.
 
 ``` erlang
--module(my_loop_handler).
--behaviour(cowboy_loop_handler).
+init(_Type, Req, _Opts) ->
+    {loop, Req, undefined_state, 30000, hibernate}.
+```
 
--export([init/3]).
--export([info/3]).
--export([terminate/3]).
+Receive loop
+------------
 
-init({tcp, http}, Req, Opts) ->
-    {loop, Req, undefined_state, 60000, hibernate}.
+Once initialized, Cowboy will wait for messages to arrive
+in the process' mailbox. When a message arrives, Cowboy
+calls the `info/3` function with the message, the Req object
+and the handler's state.
 
+The following snippet sends a reply when it receives a
+`reply` message from another process, or waits for another
+message otherwise.
+
+``` erlang
 info({reply, Body}, Req, State) ->
     {ok, Req2} = cowboy_req:reply(200, [], Body, Req),
     {ok, Req2, State};
-info(Message, Req, State) ->
+info(_Msg, Req, State) ->
     {loop, Req, State, hibernate}.
+```
+
+Do note that the `reply` tuple here may be any message
+and is simply an example.
+
+This callback may perform any necessary operation including
+sending all or parts of a reply, and will subsequently
+return a tuple indicating if more messages are to be expected.
+
+The callback may also choose to do nothing at all and just
+skip the message received.
+
+If a reply is sent, then the `ok` tuple should be returned.
+This will instruct Cowboy to end the request.
+
+Otherwise a `loop` tuple should be returned.
+
+Streaming loop
+--------------
+
+Another common case well suited for loop handlers is
+streaming data received in the form of Erlang messages.
+This can be done by initiating a chunked reply in the
+`init/3` callback and then using `cowboy_req:chunk/2`
+every time a message is received.
+
+The following snippet does exactly that. As you can see
+a chunk is sent every time a `chunk` message is received,
+and the loop is stopped by sending an `eof` message.
+
+``` erlang
+init(_Type, Req, _Opts) ->
+	{ok, Req2} = cowboy_req:chunked_reply(200, [], Req),
+    {loop, Req2, undefined_state}.
 
-terminate(Reason, Req, State) ->
-    ok.
+info(eof, Req, State) ->
+    {ok, Req, State};
+info({chunk, Chunk}, Req, State) ->
+    ok = cowboy_req:chunk(Chunk, Req),
+    {loop, Req, State};
+info(_Msg, Req, State) ->
+    {loop, Req, State}.
 ```
+
+Cleaning up
+-----------
+
+It is recommended that you set the connection header to
+`close` when replying, as this process may be reused for
+a subsequent request.
+
+Please refer to the [HTTP handlers chapter](http_handlers.md)
+for general instructions about cleaning up.
+
+Timeout
+-------
+
+By default Cowboy will not attempt to close the connection
+if there is no activity from the client. This is not always
+desirable, which is why you can set a timeout. Cowboy will
+close the connection if no data was received from the client
+after the configured time. The timeout only needs to be set
+once and can't be modified afterwards.
+
+Because the request may have had a body, or may be followed
+by another request, Cowboy is forced to buffer all data it
+receives. This data may grow to become too large though,
+so there is a configurable limit for it. The default buffer
+size is of 5000 bytes, but it may be changed by setting the
+`loop_max_buffer` middleware environment value.
+
+Hibernate
+---------
+
+To save memory, you may hibernate the process in between
+messages received. This is done by returning the atom
+`hibernate` as part of the `loop` tuple callbacks normally
+return. Just add the atom at the end and Cowboy will hibernate
+accordingly.
diff --git a/guide/middlewares.md b/guide/middlewares.md
index a1f1f8d..341a0e2 100644
--- a/guide/middlewares.md
+++ b/guide/middlewares.md
@@ -1,9 +1,6 @@
 Middlewares
 ===========
 
-Purpose
--------
-
 Cowboy delegates the request processing to middleware components.
 By default, two middlewares are defined, for the routing and handling
 of the request, as is detailed in most of this guide.
diff --git a/guide/resources.md b/guide/resources.md
deleted file mode 100644
index c6bb816..0000000
--- a/guide/resources.md
+++ /dev/null
@@ -1,17 +0,0 @@
-Resources
-=========
-
-Frameworks
-----------
-
- *  Please send a pull request!
-
-Helper libraries
-----------------
-
- *  [eventsource](https://github.com/jdavisp3/eventsource)
-
-Articles
---------
-
- *  Please send a pull request!
diff --git a/guide/rest_handlers.md b/guide/rest_handlers.md
index 87991a0..92b0bc5 100644
--- a/guide/rest_handlers.md
+++ b/guide/rest_handlers.md
@@ -10,8 +10,9 @@ The REST handler is the recommended way to handle requests.
 Initialization
 --------------
 
-Like Websocket, REST is a sub-protocol of HTTP. It therefore
-requires a protocol upgrade.
+First, the `init/3` callback is called. This callback is common
+to all handlers. To use REST for the current request, this function
+must return an `upgrade` tuple.
 
 ``` erlang
 init({tcp, http}, Req, Opts) ->
diff --git a/guide/toc.md b/guide/toc.md
index 9787f0a..b9e40ce 100644
--- a/guide/toc.md
+++ b/guide/toc.md
@@ -24,6 +24,12 @@ HTTP
  *  [Sending a response](resp.md)
  *  [Using cookies](cookies.md)
 
+Multipart
+---------
+
+ *  [Introduction to multipart](multipart_intro.md)
+ *  [Multipart requests](multipart_req.md)
+
 Static files
 ------------
 
@@ -36,48 +42,26 @@ REST
  *  [Handling REST requests](rest_handlers.md)
  *  [REST flowcharts](rest_flowcharts.md)
 
-Multipart
+Websocket
 ---------
 
- *  [Introduction to multipart](multipart_intro.md)
- *  [Multipart requests](multipart_req.md)
- *  Multipart responses
-
-Server push technologies
-------------------------
-
- *  Push technologies
- *  [Using loop handlers for server push](loop_handlers.md)
- *  CORS
-
-Using Websocket
----------------
-
  *  [The Websocket protocol](ws_protocol.md)
  *  [Handling Websocket connections](ws_handlers.md)
 
-Advanced HTTP
--------------
+Server push
+-----------
 
- *  Authentication
- *  Sessions
+ *  [Loop handlers](loop_handlers.md)
 
-Advanced Cowboy usage
----------------------
+Pluggable interface
+-------------------
 
- *  Optimization guide
- *  [Hooks](hooks.md)
  *  [Middlewares](middlewares.md)
- *  Access and error logs
- *  Handling broken clients
-   *  HTTP header names
-   *  HTTP/1.1 streaming not chunked
-
-Old guide misc
---------------
+ *  [Protocol upgrades](upgrade_protocol.md)
+ *  [Hooks](hooks.md)
 
-This section will be removed as content is moved into other chapters.
+Internals
+---------
 
- *  [Handlers](handlers.md)
- *  [Internals](internals.md)
- *  [Resources](resources.md)
+ *  [Architecture](architecture.md)
+ *  [Dealing with broken clients](broken_clients.md)
diff --git a/guide/upgrade_protocol.md b/guide/upgrade_protocol.md
new file mode 100644
index 0000000..db7a453
--- /dev/null
+++ b/guide/upgrade_protocol.md
@@ -0,0 +1,37 @@
+Protocol upgrades
+=================
+
+Cowboy features many different handlers, each for different purposes.
+All handlers have a common entry point: the `init/3` function.
+
+The default handler type is the simple HTTP handler.
+
+To switch to a different protocol, you must perform a protocol
+upgrade. This is what is done for Websocket and REST and is
+explained in details in the respective chapters.
+
+You can also create your own protocol on top of Cowboy and use
+the protocol upgrade mechanism to switch to it.
+
+For example, if you create the `my_protocol` module implementing
+the `cowboy_sub_protocol` behavior, then you can upgrade to it
+by simply returning the module name from `init/3`.
+
+``` erlang
+init(_, _, _Opts) ->
+    {upgrade, protocol, my_protocol}.
+```
+
+The `cowboy_sub_protocol` behavior only requires one callback,
+`upgrade/4`. It receives the Req object, the middleware environment,
+and the handler and options for this request. This is the same
+module as the `init/3` function and the same options that were
+passed to it.
+
+``` erlang
+upgrade(Req, Env, Handler, HandlerOpts) ->
+    %% ...
+```
+
+This callback is expected to behave like a middleware. Please
+see the corresponding chapter for more information.