Docs on Nine Nines

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Cowboy is a lightweight HTTP server.

It is built on top of Ranch. Please see the Ranch guide for more information.

One process per connection

It uses only one process per connection. The process where your code runs is the process controlling the socket. Using one process instead of two allows for lower memory usage.

Because there can be more than one request per connection with the keepalive feature of HTTP/1.1, that means the same process will be used to handle many requests.

Because of this, you are expected to make sure your process cleans up before terminating the handling of the current request. This may include cleaning up the process dictionary, timers, monitoring and more.

Binaries

It uses binaries. Binaries are more efficient than lists for representing strings because they take less memory space. Processing performance can vary depending on the operation. Binaries are known for generally getting a great boost if the code is compiled natively. Please see the HiPE documentation for more details.

Date header

Because querying for the current date and time can be expensive, Cowboy generates one Date header value every second, shares it to all other processes, which then simply copy it in the response. This allows compliance with HTTP/1.1 with no actual performance loss.

Max connections

By default the maximum number of active connections is set to a generally accepted big enough number. This is meant to prevent having too many processes performing potentially heavy work and slowing everything else down, or taking up all the memory.

Disabling this feature, by setting the {max_connections, infinity} protocol option, would give you greater performance when you are only processing short-lived requests.

AsciiDoc documentation

Mon, 01 Jan 0001 00:00:00 +0000

Erlang.mk provides rules for generating documentation from AsciiDoc files. It can automatically build a user guide PDF, chunked HTML documentation and Unix manual pages.

Requirements

It is necessary to have AsciiDoc, xsltproc and dblatex installed on your system for Erlang.mk to generate documentation from AsciiDoc sources.

Writing AsciiDoc documentation

AsciiDoc is a text document format for writing notes, documentation, articles, books, ebooks, slideshows, web pages, man pages and blogs. AsciiDoc files can be translated to many formats including HTML, PDF, EPUB, man page.

The AsciiDoc user guide describes the AsciiDoc syntax.

The Erlang.mk user guide is written in AsciiDoc and can be used as an example. The entry file is book.asciidoc.

Erlang.mk expects you to put your documentation in a specific location. This is doc/src/guide/ for the user guide, and doc/src/manual/ for the function reference. In the case of the user guide, the entry point is always doc/src/guide/book.asciidoc.

For manual pages, it is good practice to use section 3 for modules, and section 7 for the application itself.

Configuration

All of the AsciiDoc related configuration can be done directly inside the files themselves.

Usage

To build all documentation:

$ make docs

To build only the AsciiDoc documentation:

$ make asciidoc

To build only the user guide:

$ make asciidoc-guide

To build only the manual:

$ make asciidoc-manual

To install man pages on Unix:

$ make install-docs

Erlang.mk allows customizing the installation path and sections of the man pages to be installed. The MAN_INSTALL_PATH variable defines where man pages will be installed. It defaults to /usr/local/share/man. The MAN_SECTIONS variable defines which manual sections are to be installed. It defaults to 3 7.

To install man pages to a custom location:

$ make install-docs MAN_INSTALL_PATH=/opt/share/man

Note that you may need to run the install commands using sudo or equivalent if the location is not writeable by your user.

Building

Mon, 01 Jan 0001 00:00:00 +0000

Erlang.mk can do a lot of things, but it is, first and foremost, a build tool. In this chapter we will cover the basics of building a project with Erlang.mk.

For most of this chapter, we will assume that you are using a project generated by Erlang.mk.

How to build

To build a project, all you have to do is type make:

$ make

It will work regardless of your project: OTP applications, library applications, NIFs, port drivers or even releases. Erlang.mk also automatically downloads and compiles the dependencies for your project.

All this is possible thanks to a combination of configuration and conventions. Most of the conventions come from Erlang/OTP itself so any seasoned Erlang developers should feel right at home.

What to build

Erlang.mk gives you control over three steps of the build process, allowing you to do a partial build if needed.

A build has three phases: first any dependency is fetched and built, then the project itself is built and finally a release may be generated when applicable. A release is only generated for projects specifically configured to do so.

Erlang.mk handles those three phases automatically when you type make. But sometimes you just want to repeat one or two of them.

The commands detailed in this section are most useful after you have a successful build as they allow you to quickly redo a step instead of going through everything. This is especially useful for large projects or projects that end up generating releases.

Application

You can build your application and dependencies without generating a release by running the following command:

$ make app

To build your application without touching dependencies at all, you can use the SKIP_DEPS variable:

$ make app SKIP_DEPS=1

This command is very useful if you have a lot of dependencies and develop on a machine with slow file access, like the Raspberry Pi and many other embedded devices.

Note that this command may fail if a required dependency is missing.

Dependencies

You can build all dependencies, and nothing else, by running the following command:

$ make deps

This will fetch and compile all dependencies and their dependencies, recursively.

Packages and dependencies are covered in the next chapter.

Release

It is not possible to build the release without at least building the application itself, unless of course if there’s no application to begin with.

To generate the release, make will generally suffice with a normal Erlang.mk. A separate target is however available, and will take care of building the release, after building the application and all dependencies:

$ make rel

Consult the Releases chapter for more information about what releases are and how they are generated.

Application resource file

When building your application, Erlang.mk will generate the application resource file. This file is mandatory for all Erlang applications and is found in ebin/$(PROJECT).app.

PROJECT is a variable defined in your Makefile and taken from the name of the directory when Erlang.mk bootstraps your project.

Erlang.mk can build the ebin/$(PROJECT).app in two different ways: from the configuration found in the Makefile, or from the src/$(PROJECT).app.src file.

Application configuration

Erlang.mk automatically fills the PROJECT variable when bootstrapping a new project, but everything else is up to you. None of the values are required to build your project, although it is recommended to fill everything relevant to your situation.

PROJECT: The name of the OTP application or library.
PROJECT_DESCRIPTION: Short description of the project.
PROJECT_VERSION: Current version of the project.
PROJECT_REGISTERED: List of the names of all registered processes.
LOCAL_DEPS: List of Erlang/OTP applications this project depends on, excluding erts, kernel and stdlib, or list of dependencies local to this repository (in APPS_DIR).
DEPS: List of applications this project depends on that need to be fetched by Erlang.mk.

There’s no need for quotes or anything. The relevant part of the Cowboy Makefile follows, if you need an example:

PROJECT = cowboy
PROJECT_DESCRIPTION = Small, fast, modular HTTP server.
PROJECT_VERSION = 2.0.0-pre.2
PROJECT_REGISTERED = cowboy_clock

LOCAL_DEPS = crypto
DEPS = cowlib ranch

Any space before and after the value is dropped.

Dependencies are covered in details in the next chapter.

Legacy method

The src/$(PROJECT).app.src file is a legacy method of building Erlang applications. It was introduced by the original rebar build tool, of which Erlang.mk owes a great deal as it is its main inspiration.

The .app.src file serves as a template to generate the .app file. Erlang.mk will take it, fill in the modules value dynamically, and save the result in ebin/$(PROJECT).app.

When using this method, Erlang.mk cannot fill the applications key from dependencies automatically, which means you need to add them to Erlang.mk and to the .app.src at the same time, duplicating the work.

If you really can’t live without the legacy method, for one reason or another, worry not; Erlang.mk will support it. And if you need to create a new project that uses this method, you just have to say so when bootstrapping:

$ make -f erlang.mk bootstrap-lib LEGACY=1

Automatic application resource file values

When building the application resource file, Erlang.mk may automatically add an id key with information about the Git commit (if using Git), or an empty string otherwise. It will only do this under specific conditions:

The application was built as a dependency of another, or
The legacy method was used, and the .app.src file contained {id, "git"}

This value is most useful when you need to help your users, as it allows you to know which version they run exactly by asking them to look in the file, or by running a simple command on their production server:

1> application:get_all_key(cowboy).
{ok,[{description,"Small, fast, modular HTTP server."},
     {id,"2.0.0-pre.2-25-g0ffde50-dirty"},

File formats

Erlang.mk supports a variety of different source file formats. The following formats are supported natively:

Extension	Location	Description	Output
.erl	src/	Erlang source	ebin/*.beam
.core	src/	Core Erlang source	ebin/*.beam
.xrl	src/	Leex source	src/*.erl
.yrl	src/	Yecc source	src/*.erl
.asn1	asn1/	ASN.1 files	include/.hrl include/.asn1db src/*.erl
.mib	mibs/	SNMP MIB files	include/.hrl priv/mibs/.bin

Files are always searched recursively.

The build is ordered, so that files that generate Erlang source files are run before, and the resulting Erlang source files are then built normally.

In addition, Erlang.mk keeps track of header files (.hrl) as described at the end of this chapter. It can also compile C code, as described in the NIFs and port drivers chapter.

Erlang.mk also comes with plugins for the following formats:

Extension	Location	Description	Output
.dtl	templates/	Django templates	ebin/*.beam
.proto	src/	Protocol buffers	ebin/*.beam

Compilation options

Erlang.mk provides a few variables that you can use to customize the build process and the resulting files.

ERLC_OPTS

ERLC_OPTS can be used to pass some options to erlc, the Erlang compiler. Erlang.mk does not restrict any option. Please refer to the erlc Manual for the full list.

By default, Erlang.mk will set the following options:

ERLC_OPTS = -Werror +debug_info +warn_export_vars +warn_shadow_vars +warn_obsolete_guard

In other words: warnings as errors, debug info (recommended) and enable warnings for exported variables, shadow variables and obsolete guard functions.

You can redefine this variable in your Makefile to change it completely, either before or after including Erlang.mk:

ERLC_OPTS = +debug_info

You can also filter out some options from the defaults Erlang.mk sets, by defining ERLC_OPTS after including Erlang.mk using the := operator.

include erlang.mk

ERLC_OPTS := $(filter-out -Werror,$(ERLC_OPTS))

ERLC_EXCLUDE

ERLC_EXCLUDE can be used to exclude some modules from the compilation. It’s there for handling special cases, you should not normally need it.

To exclude a module, simply list it in the variable, either before or after including Erlang.mk:

ERLC_EXCLUDE = cowboy_http2

Cold and hot builds

The first time you run make, Erlang.mk will build everything.

The second time you run make, and all subsequent times, Erlang.mk will only rebuild what changed. Erlang.mk has been optimized for this use case, as it is the most common during development.

Erlang.mk figures out what changed by using the dependency tracking feature of Make. Make automatically rebuilds a target if one of its dependency has changed (for example if a header file has changed, all the source files that include it will be rebuilt), and Erlang.mk leverages this feature to cut down on rebuild times.

Note that this applies only to building; some other features of Erlang.mk will run every time they are called regardless of files changed.

Dependency tracking

Note	This section is about the dependency tracking between files inside your project, not application dependencies.

Erlang.mk keeps track of the dependencies between the different files in your project. This information is kept in the $(PROJECT).d file in your directory. It is generated if missing, and will be generated again after every file change, by default.

Dependency tracking is what allows Erlang.mk to know when to rebuild Erlang files when header files, behaviors or parse transforms have changed. Erlang.mk also automatically keeps track of which files should be compiled first, for example when you have behaviors used by other modules in your project.

If your project is stable, you may want to disable generating the dependency tracking file every time you compile. You can do this by adding the following line to your Makefile:

NO_MAKEDEP ?= 1

As you can see, the snippet above uses ?= instead of a simple equal sign. This is to allow you to temporarily override this value when you do make substantial changes to your project (including a new header file, new module with dependencies, etc.) and want to rebuild the dependency tracking file. You’ll be able to use the following command:

$ NO_MAKEDEP= make

Otherwise, make clean app will of course force the recompilation of your project.

Erlang.mk can also keep track of the source files generated by other means, for example if you generate code from a data file in your repository.

Generating Erlang source

Erlang.mk provides hooks at different stages of the build process. When your goal is to generate Erlang source files, you can add your own rules before or after the dependency tracking file is generated. To do this, you would add your hook before or after including the erlang.mk file.

The easiest way is after:

PROJECT = example

include erlang.mk

$(PROJECT).d:: src/generated_mod.erl

src/generated_mod.erl:: gen-mod.sh
        $(gen_verbose) ./gen-mod.sh $@

In this case we use $(gen_verbose) to hide the details of the build by default. Erlang.mk will simply say what file is it currently generating.

When using an external script to generate the Erlang source file, it is recommended to depend on that script, so that the source file gets generated again when the script gets modified.

If for whatever reason you prefer to hook before including Erlang.mk, don’t forget to set the .DEFAULT_GOAL variable, otherwise nothing will get built:

PROJECT = example

.DEFAULT_GOAL = all

$(PROJECT).d:: src/generated_mod.erl

include erlang.mk

src/generated_mod.erl:: gen-mod.sh
        $(gen_verbose) ./gen-mod.sh $@

Cleaning

Building typically involves creating a lot of new files. Some are reused in rebuilds, some are simply replaced. All can be removed safely.

Erlang.mk provides two commands to remove them: clean and distclean. clean removes all the intermediate files that were created as a result of building, including the BEAM files, the dependency tracking file and the generated documentation. distclean removes these and more, including the downloaded dependencies, Dialyzer’s PLT file and the generated release, putting your directory back to the state it was before you started working on it.

To clean:

$ make clean

Or distclean:

$ make distclean

That is the question.

Note that Erlang.mk will automatically clean some files as part of other targets, but it will never run distclean if you don’t explicitly use it.

Code coverage

Mon, 01 Jan 0001 00:00:00 +0000

Placeholder chapter.

Common Test

Mon, 01 Jan 0001 00:00:00 +0000

Common Test is Erlang’s functional testing framework. Erlang.mk automates the discovery and running of Common Test suites.

Writing tests

The Common Test user guide is the best place to learn how to write tests. Erlang.mk requires that file names for test suites end with _SUITE.erl and that the files be located in the $(TEST_DIR) directory. This defaults to test/.

Configuration

The CT_OPTS variable allows you to set extra Common Test options. Options are documented in the Common Test user guide. You can use it to set Common Test hooks, for example:

CT_OPTS = -ct_hooks cowboy_ct_hook

The CT_SUITES variable can be used to override what Common Test suites Erlang.mk will be aware of. It does not normally need to be set as Erlang.mk will find the test suites automatically.

The name of the suite is the part before _SUITE.erl. If the file is named http_SUITE.erl, the test suite is http:

CT_SUITES = http ws

Usage

To run all tests (including Common Test):

$ make tests

To run all tests and static checks (including Common Test):

$ make check

You can also run Common Test separately:

$ make ct

Erlang.mk will create targets for all test suites it finds. If you have a file named test/http_SUITE.erl, then the target ct-http will run that specific test suite:

$ make ct-http

Erlang.mk provides a convenient way to run a specific group or a specific test case within a specific group, using the variable t. Note that this only applies to suite-specific targets, like the ct-http example above.

To run all tests from the http_compress group in the http_SUITE test suite, write:

$ make ct-http t=http_compress

Similarly, to run a specific test case in that group:

$ make ct-http t=http_compress:headers_dupe

To do the same against a multi-application repository, you can use the -C option:

$ make -C apps/my_app ct-http t=my_group:my_case

Note that this also applies to dependencies. When using Cowboy as a dependency, you can run the following directly:

$ make -C deps/cowboy ct-http t=http_compress

Finally, code coverage is available, but covered in its own chapter.

Compatibility with other build tools

Mon, 01 Jan 0001 00:00:00 +0000

Erlang.mk tries its best to be compatible with the other Erlang build tools. It can use dependencies written with other build tools in mind, and can also make your projects usable by those build tools as well. Erlang.mk is like the cool kid that gets along with everybody.

In this chapter I will use the term Rebar project to refer to a project built using Rebar 2, Rebar 3 or Mad. These three build tools are very similar and share the same configuration file.

Rebar projects as Erlang.mk dependencies

Erlang.mk comes with a feature called Autoload which will use Rebar 2 to patch any Rebar project and make it compatible with Erlang.mk. This feature essentially patches Rebar out and adds a Makefile to the project that Erlang.mk can then use for building:

Autoload is documented in more details in the Packages and dependencies chapter.

Erlang.mk projects as Rebar dependencies

Erlang.mk projects can be made compatible with the Rebar family of build tools pretty easily, as Erlang.mk will generate all the files they require for building.

The Rebar family requires two files: a rebar.config file containing compilation options and the list of dependencies, and the application resource file, found either at ebin/$(PROJECT).app or at src/$(PROJECT).app.src.

Rebar configuration

Erlang.mk comes with a target that generates a rebar.config file when invoked:

$ make rebar.config

Careful! This will build the file even if it already existed before.

To build this file, Erlang.mk uses information it finds in the DEPS and ERLC_OPTS variables, among others. This means that the Rebar family builds your project much the same way as Erlang.mk.

Careful though! Different build tools have different fetching strategies. If some applications provide differing dependencies, they might be fetched differently by other build tools. Check the upcoming Sanity check chapter to find out how to detect such issues.

You can automatically generate this file when you build your application, by making it a dependency of the app target:

app:: rebar.config

Don’t forget to commit the file when it changes!

If you run into other issues, it’s probably because you use a feature specific to Erlang.mk, like the cp fetch method. It could also be that we forgot to handle something! Sorry. We are of course interested to hear about any compatibility problems you may have, just open a ticket!

Application resource file

Erlang.mk has two ways to generate an application resource file: from the information found in the Makefile, or from the information found in the src/$(PROJECT).app.src file. Needless to say, if you have this file in your repository, then you don’t need to worry about compatibility with other build tools.

If you don’t, however, it’s not much harder. Every time Erlang.mk will compile your application, it will produce a new ebin/$(PROJECT).app file. Simply commit this file when it changes. It will only change when you modify the configuration, add or remove modules.

Connection

Mon, 01 Jan 0001 00:00:00 +0000

This chapter describes how to open, monitor and close a connection using the Gun client.

Gun connections

Gun is designed with the SPDY and Websocket protocols in mind. They are built for long-running connections that allow concurrent exchange of data, either in the form of request/responses for SPDY or in the form of messages for Websocket.

A Gun connection is an Erlang process that manages a socket to a remote endpoint. This Gun connection is owned by a user process that is called the owner of the connection, and is managed by the supervision tree of the gun application.

The owner process communicates with the Gun connection by calling functions from the module gun. All functions perform their respective operations asynchronously. The Gun connection will send Erlang messages to the owner process whenever needed.

When the remote endpoint closes the connection, Gun attempts to reconnect automatically.

Opening a new connection

The gun:open/{2,3} function must be used to open a connection.

Opening a connection to example.org on port 443

{ok, ConnPid} = gun:open("example.org", 443).

If the port given is 443, Gun will attempt to connect using SSL. The protocol will be selected automatically using the NPN extension for TLS. By default Gun supports SPDY/3.1, SPDY/3 and HTTP/1.1 when connecting using SSL.

For any other port, Gun will attempt to connect using TCP and will use the HTTP/1.1 protocol.

The transport and protocol used can be overriden using options. The manual documents all available options.

Options can be provided as a third argument, and take the form of a map.

Opening an SSL connection to example.org on port 8443

{ok, ConnPid} = gun:open("example.org", 8443, #{transport=>ssl}).

Waiting for the connection to be established

When Gun successfully connects to the server, it sends a gun_up message with the protocol that has been selected for the connection.

Gun provides the functions gun:await_up/{1,2,3} that wait for the gun_up message. They can optionally take a monitor reference and/or timeout value. If no monitor is provided, one will be created for the duration of the function call.

Synchronous opening of a connection

{ok, ConnPid} = gun:open("example.org", 443),
{ok, Protocol} = gun:await_up(ConnPid).

Handling connection loss

When the connection is lost, Gun will send a gun_down message indicating the current protocol, the reason the connection was lost and two list of stream references.

The first list indicates open streams that may have been processed by the server. The second list indicates open streams that the server did not process.

Monitoring the connection process

@todo Gun should detect the owner process being killed

Because software errors are unavoidable, it is important to detect when the Gun process crashes. It is also important to detect when it exits normally. Erlang provides two ways to do that: links and monitors.

Gun leaves you the choice as to which one will be used. However, if you use the gun:await/{2,3} or gun:await_body/{2,3} functions, a monitor may be used for you to avoid getting stuck waiting for a message that will never come.

If you choose to monitor yourself you can do it on a permanent basis rather than on every message you will receive, saving resources. Indeed, the gun:await/{3,4} and gun:await_body/{3,4} functions both accept a monitor argument if you have one already.

Monitoring the connection process

{ok, ConnPid} = gun:open("example.org", 443).
MRef = monitor(process, ConnPid).

This monitor reference can be kept and used until the connection process exits.

Handling DOWN messages

receive
        %% Receive Gun messages here...
        {'DOWN', Mref, process, ConnPid, Reason} ->
                error_logger:error_msg("Oops!"),
                exit(Reason);
end.

What to do when you receive a DOWN message is entirely up to you.

Closing the connection abruptly

The connection can be stopped abruptly at any time by calling the gun:close/1 function.

Immediate closing of the connection

gun:close(ConnPid).

The process is stopped immediately without having a chance to perform the protocol’s closing handshake, if any.

Closing the connection gracefully

The connection can also be stopped gracefully by calling the gun:shutdown/1 function.

Graceful shutdown of the connection

gun:shutdown(ConnPid).

Gun will refuse any new requests or messages after you call this function. It will however continue to send you messages for existing streams until they are all completed.

For example if you performed a GET request just before calling gun:shutdown/1, you will still receive the response before Gun closes the connection.

If you set a monitor beforehand, you will receive a message when the connection has been closed.

Constraints

Mon, 01 Jan 0001 00:00:00 +0000

Cowboy provides an optional constraints based validation feature when interacting with user input.

Constraints are first used during routing. The router uses constraints to more accurately match bound values, allowing to create routes where a segment is an integer for example, and rejecting the others.

Constraints are also used when performing a match operation on input data, like the query string or cookies. There, a default value can also be provided for optional values.

Finally, constraints can be used to not only validate input, but also convert said input into proper Erlang terms, all in one step.

Structure

Constraints are provided as a list of fields and for each field a list of constraints for that field can be provided.

Fields are either the name of the field; the name and one or more constraints; or the name, one or more constraints and a default value.

When no default value is provided then the field is required. Otherwise the default value is used.

All constraints for a field will be used to match its value in the order they are given. If the value is modified by a constraint, the next constraint receives the updated value.

Built-in constraints

Constraint	Description
int	Convert binary value to integer.
nonempty	Ensures the binary value is non-empty.

Custom constraint

In addition to the predefined constraints, Cowboy will accept a fun. This fun must accept one argument and return one of true, {true, NewValue} or false. The result indicates whether the value matches the constraint, and if it does it can optionally be modified. This allows converting the value to a more appropriate Erlang term.

Note that constraint functions SHOULD be pure and MUST NOT crash.

Continuous integration

Mon, 01 Jan 0001 00:00:00 +0000

Placeholder chapter.

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

You are welcome and encouraged to contribute.

This is how.

Priorities

From the most important to the least important:

Bugs
Package issues/additions
Refactoring
Features

Bugs

If you have found a bug, you should open a ticket. Include everything relevant including the command you used, output, a link to the code that triggers the issue, why you think this is a bug, etc.

If you think you have found a bug but you are not sure, you should open a ticket as previously explained.

If you have found a bug and you need it to be solved RIGHT NOW, open a ticket as previously explained.

Once you have opened a ticket, be patient, try to answer questions in a timely manner and confirm that the bug was indeed fixed when it is.

If you can’t be patient, either try to solve the bug and contribute the fix back or become a paying customer.

Code

The code is located in the core/*.mk and plugins/*.mk files. The tests are located in the test/Makefile and test/*.mk files.

If you have a fix or a hack for a bug, you should open a pull request. Any fix should include a test case that fails before the fix and is working after.

If you have a test case that reproduces a bug, but no fix for it, you should open a pull request.

Changes need to be tested with at least the make check command. A specific test case can be tested using make check c=CASE with CASE the name of the target to run. Output can be modulated using the V variable, which is an integer from 0 to 4. A typical use would be make check c=dialyzer V=3. The value 4 is particular and shows expanded commands right before they are executed.

To run tests in parallel, use the -j option. It is generally a good idea to also use the -k option to run all tests even if one fails. For example: make check -j 32 -k.

Some changes should be tested against all packages. Continue reading for more details on testing them.

Packages

You can search existing packages using the make search q=STRING command. This can be done both from an Erlang.mk project or directly from the Erlang.mk repository.

Packages can be added to the index using the pkg_add.sh script.

$ git clone https://github.com/$YOURUSERNAME/erlang.mk
$ cd erlang.mk
$ ./pkg_add.sh cowboy git https://github.com/ninenines/cowboy 1.0.0
  http://ninenines.eu "Small, fast and modular HTTP server."
$ git push origin master

Before sending a pull request, you should test your package. You can use the following command: make check p=PACKAGE, where PACKAGE is the name of the package, for example cowboy.

To test all packages, the make packages command can be used. This can take a long time. Some packages will fail with certain versions of Erlang, or if a prerequisite is missing from your system. You can of course speed things up using the -j and -k flags.

After all packages have been tested, you can run the command make summary to know what changed since the previous run.

Documentation

The documentation is always right.

If you think you have found a mistake in the documentation, this is a bug. You can either open a ticket or send a pull request.

To make sure that the documentation changes work, install the listed Requirements on your system and run make docs.

Feature requests

If you have an awesome idea or need something that Erlang.mk doesn’t provide yet, open a ticket. Provide as much detail as possible.

If you have code, great! Open a pull request as previously explained.

If not, you can still improve your feature request by writing the related documentation.

Cowboy Function Reference

Mon, 01 Jan 0001 00:00:00 +0000

Cowboy User Guide

Mon, 01 Jan 0001 00:00:00 +0000

Rationale

Introduction

Configuration

Request and response

REST

Websocket

Internals

Dealing with broken clients

Mon, 01 Jan 0001 00:00:00 +0000

There exists a very large number of implementations for the HTTP protocol. Most widely used clients, like browsers, follow the standard quite well, but others may not. In particular custom enterprise clients tend to be very badly written.

Cowboy tries to follow the standard as much as possible, but is not trying to handle every possible special cases. Instead Cowboy focuses on the cases reported in the wild, on the public Web.

That means clients that ignore the HTTP standard completely may fail to understand Cowboy’s responses. There are of course workarounds. This chapter aims to cover them.

Lowercase headers

Cowboy converts all headers it receives to lowercase, and similarly sends back headers all in lowercase. Some broken HTTP clients have issues with that.

A simple way to solve this is to create an onresponse hook that will format the header names with the expected case.

capitalize_hook(Status, Headers, Body, Req) ->
    Headers2 = [{cowboy_bstr:capitalize_token(N), V}
        || {N, V} <- Headers],
    cowboy_req:reply(Status, Headers2, Body, Req).

Note that HTTP/2 clients do not have that particular issue because the specification explicitly says all headers are lowercase, unlike HTTP which allows any case but treats them as case insensitive.

Camel-case headers

Sometimes it is desirable to keep the actual case used by clients, for example when acting as a proxy between two broken implementations. There is no easy solution for this other than forking the project and editing the cowboy_protocol file directly.

Chunked transfer-encoding

Sometimes an HTTP client advertises itself as HTTP/1.1 but does not support chunked transfer-encoding. This is invalid behavior, as HTTP/1.1 clients are required to support it.

A simple workaround exists in these cases. By changing the Req object response state to waiting_stream, Cowboy will understand that it must use the identity transfer-encoding when replying, just like if it was an HTTP/1.0 client.

Req2 = cowboy_req:set(resp_state, waiting_stream).

Designing a resource handler

Mon, 01 Jan 0001 00:00:00 +0000

This chapter aims to provide you with a list of questions you must answer in order to write a good resource handler. It is meant to be usable as a step by step guide.

The service

Can the service become unavailable, and when it does, can we detect it? For example, database connectivity problems may be detected early. We may also have planned outages of all or parts of the system. Implement the service_available callback.

What HTTP methods does the service implement? Do we need more than the standard OPTIONS, HEAD, GET, PUT, POST, PATCH and DELETE? Are we not using one of those at all? Implement the known_methods callback.

Type of resource handler

Am I writing a handler for a collection of resources, or for a single resource?

The semantics for each of these are quite different. You should not mix collection and single resource in the same handler.

Collection handler

Skip this section if you are not doing a collection.

Is the collection hardcoded or dynamic? For example, if you use the route /users for the collection of users then the collection is hardcoded; if you use /forums/:category for the collection of threads then it isn’t. When the collection is hardcoded you can safely assume the resource always exists.

What methods should I implement?

OPTIONS is used to get some information about the collection. It is recommended to allow it even if you do not implement it, as Cowboy has a default implementation built-in.

HEAD and GET are used to retrieve the collection. If you allow GET, also allow HEAD as there’s no extra work required to make it work.

POST is used to create a new resource inside the collection. Creating a resource by using POST on the collection is useful when resources may be created before knowing their URI, usually because parts of it are generated dynamically. A common case is some kind of auto incremented integer identifier.

The next methods are more rarely allowed.

PUT is used to create a new collection (when the collection isn’t hardcoded), or replace the entire collection.

DELETE is used to delete the entire collection.

PATCH is used to modify the collection using instructions given in the request body. A PATCH operation is atomic. The PATCH operation may be used for such things as reordering; adding, modifying or deleting parts of the collection.

Single resource handler

Skip this section if you are doing a collection.

What methods should I implement?

OPTIONS is used to get some information about the resource. It is recommended to allow it even if you do not implement it, as Cowboy has a default implementation built-in.

HEAD and GET are used to retrieve the resource. If you allow GET, also allow HEAD as there’s no extra work required to make it work.

POST is used to update the resource.

PUT is used to create a new resource (when it doesn’t already exist) or replace the resource.

DELETE is used to delete the resource.

PATCH is used to modify the resource using instructions given in the request body. A PATCH operation is atomic. The PATCH operation may be used for adding, removing or modifying specific values in the resource.

The resource

Following the above discussion, implement the allowed_methods callback.

Does the resource always exist? If it may not, implement the resource_exists callback.

Do I need to authenticate the client before they can access the resource? What authentication mechanisms should I provide? This may include form-based, token-based (in the URL or a cookie), HTTP basic, HTTP digest, SSL certificate or any other form of authentication. Implement the is_authorized callback.

Do I need fine-grained access control? How do I determine that they are authorized access? Handle that in your is_authorized callback.

Can access to a resource be forbidden regardless of access being authorized? A simple example of that is censorship of a resource. Implement the forbidden callback.

Are there any constraints on the length of the resource URI? For example, the URI may be used as a key in storage and may have a limit in length. Implement uri_too_long.

Representations

What media types do I provide? If text based, what charsets are provided? What languages do I provide?

Implement the mandatory content_types_provided. Prefix the callbacks with to_ for clarity. For example, to_html or to_text.

Implement the languages_provided or charsets_provided callbacks if applicable.

Is there any other header that may make the representation of the resource vary? Implement the variances callback.

Depending on your choices for caching content, you may want to implement one or more of the generate_etag, last_modified and expires callbacks.

Do I want the user or user agent to actively choose a representation available? Send a list of available representations in the response body and implement the multiple_choices callback.

Redirections

Do I need to keep track of what resources were deleted? For example, you may have a mechanism where moving a resource leaves a redirect link to its new location. Implement the previously_existed callback.

Was the resource moved, and is the move temporary? If it is explicitly temporary, for example due to maintenance, implement the moved_temporarily callback. Otherwise, implement the moved_permanently callback.

The request

Do we need to perform extra checks to make sure the request is valid? Cowboy will do many checks when receiving the request already, do we need more? Note that this only applies to the request-line and headers of the request, and not the body. Implement malformed_request.

May there be a request body? Will I know its size? What’s the maximum size of the request body I’m willing to accept? Implement valid_entity_length.

Finally, take a look at the sections corresponding to the methods you are implementing.

OPTIONS method

Cowboy by default will send back a list of allowed methods. Do I need to add more information to the response? Implement the options method.

GET and HEAD methods

If you implement the methods GET and/or HEAD, you must implement one ProvideResource callback for each content-type returned by the content_types_provided callback.

PUT, POST and PATCH methods

If you implement the methods PUT, POST and/or PATCH, you must implement the content_types_accepted callback, and one AcceptResource callback for each content-type it returns. Prefix the AcceptResource callback names with from_ for clarity. For example, from_html or from_json.

Do we want to allow the POST method to create individual resources directly through their URI (like PUT)? Implement the allow_missing_post callback. It is recommended to explicitly use PUT in these cases instead.

May there be conflicts when using PUT to create or replace a resource? Do we want to make sure that two updates around the same time are not cancelling one another? Implement the is_conflict callback.

DELETE methods

If you implement the method DELETE, you must implement the delete_resource callback.

When delete_resource returns, is the resource completely removed from the server, including from any caching service? If not, and/or if the deletion is asynchronous and we have no way of knowing it has been completed yet, implement the delete_completed callback.

Dialyzer

Mon, 01 Jan 0001 00:00:00 +0000

Dialyzer is a tool that will detect discrepancies in your program. It does so using a technique known as success typing analysis which has the advantage of providing no false positives. Dialyzer is able to detect type errors, dead code and more.

Erlang.mk provides a wrapper around Dialyzer.

How it works

Dialyzer requires a PLT file to work. The PLT file contains the analysis information from all applications which are not expected to change, or rarely do. These would be all the dependencies of the application or applications you are currently working on, including standard applications in Erlang/OTP itself.

Dialyzer can generate this PLT file. Erlang.mk includes rules to automatically generate the PLT file when it is missing.

Once the PLT file is generated, Dialyzer can perform the analysis in record time.

Configuration

In a typical usage scenario, no variable needs to be set. The defaults should be enough. Do note however that the dependencies need to be set properly using the DEPS and LOCAL_DEPS variables.

The DIALYZER_PLT file indicates where the PLT file will be written to (and read from). By default this is $(PROJECT).plt in the project’s directory. Note that the DIALYZER_PLT variable is exported and is understood by Dialyzer directly.

The PLT_APPS variable can be used to add additional applications to the PLT. You can either list application names or paths to these applications.

Erlang.mk defines two variables for specifying options for the analysis: DIALYZER_DIRS and DIALYZER_OPTS. The former one defines which directories should be part of the analysis. The latter defines what extra warnings Dialyzer should report.

Note that Erlang.mk enables the race condition warnings by default. As it can take considerably large resources to run, you may want to disable it on larger projects.

Usage

To perform an analysis, run the following command:

$ make dialyze

This will create the PLT file if it doesn’t exist.

The analysis will also be performed when you run the following command, alongside tests:

$ make check

You can use the plt target to create the PLT file if it doesn’t exist. This is normally not necessary as Dialyzer creates it automatically.

The PLT file will be removed when you run make distclean.