path: root/doc/src/guide/deps.asciidoc

[[deps]]
== Packages and dependencies

Erlang.mk can fetch and compile the dependencies that your
project requires. Erlang.mk improves upon the concepts
introduced by Rebar, so they should be familiar to many
seasoned Erlang developers.

Erlang.mk is not a package manager, nor is it trying to be,
but it does include an index of Erlang packages to make
discovering useful projects easier.

This chapter will explain how to use packages, add
dependencies to your project or bundle them directly
in a single repository.

=== Searching packages

Erlang.mk gives you access to nearly 500 packages, with more
being added regularly.

To find a package, search for it:

[source,bash]
$ make search q=pool

This will return all packages matching this word, like worker
pool and acceptor pool projects.

You can also list everything and use regular command line
tools to find what you need, for example:

[source,bash]
$ make search | less

// @todo Simplify adding packages, add a new chapter explaining
// everything, then link to this new chapter from here.

=== Adding dependencies to your project

Once you find the package you need, adding it as a dependency
to your project is a one-liner:

[source,make]
DEPS = cowboy

And that's it! The next time you run `make`, Erlang.mk will
fetch and compile Cowboy. Erlang.mk will also ensure Cowboy
is available whenever you use the shell, run tests and any
other operations.

Erlang.mk will fill in the application resource file with
all applications found in `DEPS`. But not all dependencies
are Erlang applications, and not all dependencies need to
be a runtime dependency. That's where the `BUILD_DEPS`
variable comes in: it works just like `DEPS`, except the
dependencies listed there will not be added as runtime
dependencies.

For example, you could add a parse transform project like
this to make it available only at build time:

[source,make]
BUILD_DEPS = erlando

Or you could depend on a C project directly, if you are
building a NIF:

[source,make]
BUILD_DEPS = leveldb
dep_leveldb = git https://github.com/basho/leveldb 2.1.3

This dependency will be built before your application, so
you could easily copy the resulting shared file into your
'priv/' directory as part of the build process. More information
about that in the xref:ports[NIFs and port drivers]
chapter.

Another variable, `LOCAL_DEPS`, allows specifying runtime
dependencies which are part of Erlang/OTP itself, but also
dependencies that are included in the repository. Since they
are already on your system, there is no need to fetch them.
Do note that there is no way to choose the version, the
application used will be the one already on your system.

You could depend on the Crypto application, for example:

[source,make]
LOCAL_DEPS = crypto

Erlang.mk comes with additional types of dependencies.
It has `TEST_DEPS` for dependencies used only for testing:

[source,make]
TEST_DEPS = ct_helper
dep_ct_helper = git https://github.com/ninenines/ct_helper master

`DOC_DEPS` for dependencies used only when building documentation:

[source,make]
DOC_DEPS = edown

`REL_DEPS` for dependencies required to build the release,
or to include extra applications in the release:

[source,make]
REL_DEPS = recon

And `SHELL_DEPS` for dependencies to make available when running
the `make shell` command:

[source,make]
SHELL_DEPS = tddreloader

All these will be documented in more details in their respective
chapters.

Note that these additional types of dependencies will only
be fetched after normal dependencies by default. You can
force a dependency to be downloaded first by extending the
target `deps::` before including 'erlang.mk', for example:

[source,make]
deps:: $(CURDIR)/deps/triq

==== Modifying the dependency source or version

By default, Erlang.mk will look into its package index to
find the project you are looking for, if you only provide
its name. This is this case:

[source,make]
DEPS = cowboy

If you need a different version, you need to define another
variable. There are two ways to do this, each being useful
for different reasons.

If you simply want to change the commit number, all you
need to do is to define the `dep_$(DEP_NAME)_commit`
variable. In the case of Cowboy, this would look like this:

[source,make]
DEPS = cowboy
dep_cowboy_commit = 2.0.0-pre.2

Erlang.mk will use the package index to get all information
about Cowboy, except the commit number which will be overriden.

If you need to set the fetch method or repository information
too, for example because you want to use your own fork, or
simply because the project is missing from the index, you
can define the `dep_$(DEP_NAME)` variable with everything:

[source,make]
DEPS = cowboy
dep_cowboy = git https://github.com/essen/cowboy 2.0.0-pre.2

This will fetch Cowboy from your fork at the given commit.

==== Fetch methods

Erlang.mk comes with a number of different fetch methods.
You can fetch from Git, Mercurial, SVN, to name a few.
There are fetch methods that will work everywhere, and
fetch methods that will only work in a given environment.

The following table lists all existing methods:

[cols="<,2*^",options="header"]
|===
| Name           | Format            | Description
| git            | git repo commit   | Clone the Git repository and checkout the given version
| git-subfolder  | git repo commit subfolder | Clone the Git repository, checkout the given version and use one of its subfolders as a dependency
| git-submodule  | git-submodule     | Initialize and update the Git submodule
| hg             | hg repo commit    | Clone the Mercurial repository and update to the given version
| svn            | svn repo          | Checkout the given SVN repository
| cp             | cp path/to/repo   | Recursively copy a local directory
| ln             | ln path/to/repo   | Symbolically link a local directory
| hex            | hex version [pkg] | Download the given project version from hex.pm
| fail           | N/A               | Always fail, reserved for internal use
| legacy         | N/A               | Legacy Erlang.mk fetcher, reserved for internal use
| default        | N/A               | Reserved
|===

The `git` and `hg` methods both have a repository and commit.
You can use any valid commit, tag or branch in that repository
for the commit value.

For example, to fetch Cowboy with tag 2.0.0-pre.2 from Git:

[source,make]
dep_cowboy = git https://github.com/ninenines/cowboy 2.0.0-pre.2

Or to fetch Ehsa tag 4.0.3 from Mercurial:

[source,make]
dep_ehsa = hg https://bitbucket.org/a12n/ehsa 4.0.3

Git also comes with a concept of submodules. Erlang.mk can
automatically initializes and updates submodules for dependencies,
as long as they were added beforehand using `git submodule add`:

[source,make]
dep_cowboy = git-submodule

The `svn` method only has a repository value, but that's
simply because the SVN repository URL can also contain
the path and commit.

This would fetch an example project from the trunk:

[source,make]
dep_ex1 = svn https://example.com/svn/trunk/project/ex1

And this would fetch a separate example project from a
specific commit:

[source,make]
dep_ex2 = svn svn://example.com/svn/branches/erlang-proj/ex2@264

You can copy a directory from your machine using the `cp` method.
It only takes the path to copy from:

[source,make]
dep_cowboy = cp $(HOME)/ninenines/cowboy

Finally, you can use a package from the
link:https://hex.pm/[Hex repository]:

[source,make]
dep_cowboy = hex 1.0.3

If the package on Hex has a different name than the application,
you can provide it after the version:

[source,make]
dep_uuid = hex 1.7.5 uuid_erl

==== Custom fetch methods

If none of the existing methods fit your use, you can simply
define your own. Erlang.mk will consider all variables that
are named as `dep_fetch_$(METHOD)` to be available fetch
methods. You can do anything inside this variable, as long
as you create a folder named '$(DEPS_DIR)/$(call dep_name,$1)'.
Or in layman terms, if your dependency is Cowboy, this would
become 'deps/cowboy'.

To give an example, this is what the Git method does:

[source,make]
----
define dep_fetch_git
	git clone -q -n -- $(call dep_repo,$1) $(DEPS_DIR)/$(call dep_name,$1); \
	cd $(DEPS_DIR)/$(call dep_name,$1) && git checkout -q $(call dep_commit,$1);
endef
----

Note that, like dependency information, this custom fetch method
must be written before including 'erlang.mk'.

=== How deps are fetched and built

The order in which dependencies are fetched and built is well
defined. This means that Erlang.mk will get the same applications
regardless of the command or options being used.

In tree traversal terms, where the list of dependencies is a
tree, Erlang.mk fetches everything using the pre-order traversal
method. The steps can be summarized like this, starting from
the root application:

. Fetch all dependencies for the application
. Build first dependency
. Build Nth dependency
. Build last dependency

Every time a dependency is built, these same steps are followed,
recursively.

Do note that the first step, fetching all dependencies of
an application, is not guaranteed to be ordered. The reason
for this is that it is not possible to have the same dependency
listed twice in a single application, and therefore there can
be no conflicts. Remember, this step only fetches, at no point
are different applications built in parallel.

What about conflicts between the dependencies of different
applications? Simple. Since builds are ordered, this means
that the first version of an application that is fetched
will be the one that wins.

This means that if project A depends on projects B and C,
in this order, and that both B and C depend on a different
version of D, it will always be B's version of D that wins,
because we fetch the dependencies of B before fetching
those from C.

Similarly, if project A depends on projects B, C and D,
regardless of the order, and A, B and C depend on a
different version of D, it will always be A's version
that wins, because we fetch all dependencies of A before
fetching those from B or C.

Once a dependency is built, it will not be built again by
default. Typically dependencies do not need to be recompiled
and this speeds up building immensely. There are a few ways
to force recompiling a dependency however:

* The dependency directory is a symbolic link; the dependency
  will always be recompiled.

* The dependency is built directly, for example with a command
  like `make -C deps/cowlib`, or `make` in the dependency's
  directory.

* The variable `FULL` is set, for example `make FULL=1`. This
  will force building of all dependencies. This can be added
  to your Makefile before including 'erlang.mk'.

* The file `ebin/dep_built` in the dependency is removed.

=== Fetching and listing dependencies only

You can fetch all dependencies recursively without building anything,
with the `make fetch-deps` command. It follows the same rules described
in the section above.

You can list all dependencies recursively, again without building
anything, with the `make list-deps` command. It will obviously need
to fetch all dependencies exactly like `make fetch-deps`. Once
everything is fetched, it prints a sorted list of absolute paths to the
dependencies.

By default, `fetch-deps` and `list-deps` work on the `BUILD_DEPS`
and `DEPS` lists only. To also fetch/list `TEST_DEPS`, `DOC_DEPS`,
`REL_DEPS` and/or `SHELL_DEPS`, you have two possibilities:

* You can use `make fetch-test-deps`, `make fetch-doc-deps`, `make
  fetch-rel-deps` and `make fetch-shell-deps` commands respectively.
  If you want to list them, you can use `make list-test-deps`, `make
  list-doc-deps`, `make list-rel-deps` and `make list-shell-deps`
  respectively.
* You can use `make fetch-deps` or `make list-deps` with the Makefile
  variable `DEP_TYPES` set to a list of dependency types you want.
  The types are `test`, `doc`, `rel` and `shell` respectively. For
  example, you can list test and doc dependencies with `make list-deps
  DEP_TYPES='test doc'`.

Note that only first level `TEST_DEPS`, `DOC_DEPS`, `REL_DEPS` and
`SHELL_DEPS` are included, not dependencies' one. In other word,
`make list-test-deps` lists the `TEST_DEPS` of your project, but not
`TEST_DEPS` of the projects yours depend on.

No matter which method you use, `BUILD_DEPS` and `DEPS` are always
included.

Internally, the `make fetch-*` commands store the complete list of
dependencies in files named `$(ERLANG_MK_RECURSIVE_DEPS_LIST)`,
`$(ERLANG_MK_RECURSIVE_TEST_DEPS_LIST)`,
`$(ERLANG_MK_RECURSIVE_DOC_DEPS_LIST)`,
`$(ERLANG_MK_RECURSIVE_REL_DEPS_LIST)` and
`$(ERLANG_MK_RECURSIVE_SHELL_DEPS_LIST)`. Those files are simply printed
by the `make list-*` commands.

`make list-*` commands are made for human beings. If you need the list
of dependencies in a Makefile or a script, you should use the content
of those files directly instead. The reason is that `make fetch-*` and
`make list-*` may have unwanted content in their output, such as actual
fetching of dependencies.

=== Querying dependencies

You can obtain information about all dependencies with
the `make query-deps` family of commands:

* `make query-deps` will list dependencies found in `BUILD_DEPS`
  and `DEPS` recursively.
* `make query-doc-deps` will list documentation dependencies
  of the current project.
* `make query-rel-deps` will list release dependencies
  of the current project.
* `make query-shell-deps` will list shell dependencies
  of the current project.
* `make query-test-deps` will list test dependencies
  of the current project.

By default the information printed will be the dependency name,
fetch method, repository and version, prefixed by the current
project's name. But this output can be customized via the
variable `QUERY`:

[source,bash]
$ make query-deps QUERY="name fetch_method repo version extra absolute_path"

The following options are available:

name:: The dependency name.
fetch_method:: The dependency's fetch method.
repo:: The dependency's repository.
version:: The dependency's version, tag or commit.
extra:: Any additional information specific to the fetch method used.
absolute_path:: The dependency's location after it has been fetched.

Fields that have no value will print `-`. For example
not all fetch methods have a value for the `version`.

The value for `extra`, when available, will be formatted
with the name of the information printed prefixed. For
example the hex fetch method will add
`package-name=uuid_erl` for the `uuid` application.

=== Ignoring unwanted dependencies

Sometimes, you may want to ignore dependencies entirely.
Not even fetch them. You may want to do this because a
project you depend on depends on an application you do
not need (like a dependency for building documentation
or testing). Or maybe the dependency is already installed
on your system.

To ignore a dependency, simply add it to the `IGNORE_DEPS`
variable:

[source,make]
IGNORE_DEPS += edown proper

This will only ignore dependencies that are needed for
building. It is therefore safe to write:

[source,make]
IGNORE_DEPS += edown proper
TEST_DEPS = proper

The PropEr application will be fetched as intended when
running `make tests` or `make check`. It will however
not be fetched when running `make` or `make deps`.

=== Dependencies directory

Dependencies are fetched in '$(DEPS_DIR)'. By default this is
the 'deps' directory. You can change this default, but you
should only do so if it was not defined previously. Erlang.mk
uses this variable to tell dependencies where to fetch their
own dependencies.

You will therefore need to use `?=` instead of `=`. Of course,
if you know you will never use this project as a dependency,
`=` will work. But to avoid it biting you later on, do this:

[source,make]
DEPS_DIR ?= $(CURDIR)/libs

The `$(CURDIR)` part is important, otherwise dependencies of
dependencies will be fetched in the wrong directory.

Erlang.mk will also export the `REBAR_DEPS_DIR` variable for
compatibility with Rebar build tools, as long as they are
recent enough.

=== Many applications in one repository

In addition to the dependencies that are fetched, Erlang.mk
also allows you to have dependencies local to your repository.
This kind of layout is sometimes called multi-application
repositories, or repositories with multiple applications.

They work exactly the same as remote dependencies, except:

* They are not fetched
* They are not autopatched
* They are not deleted on `make distclean`
* They are not automatically added to the application resource file

To properly fill the application resource file and compile apps in
the right order, you will need to define the `LOCAL_DEPS` variable
for each relevant application, the same as for OTP applications. Apps
can depend on each other in this way, and their compilation order
will follow the same rules as regular dependencies in `DEPS`.

The top-level `LOCAL_DEPS` variable, if defined, will determine which
apps (along with their dependencies) to build, and also which apps
should be added to the top-level application resource file, if there
is one. This may be useful, for example, for specifying a different
set of apps to build for different releases. If `LOCAL_DEPS` is not
defined, then all apps in the '$(APPS_DIR)' will be built, but none
will be automatically added to the top-level application resource
file.

If there is a conflict between a local dependency and a
remote dependency, then the local dependency always wins;
an error will be triggered when trying to fetch the
conflicting remote dependency.

To start using dependencies local to the repository, simply
create a folder named '$(APPS_DIR)'. By default, this folder
is the 'apps/' directory.

You can use Erlang.mk to bootstrap local dependencies by
using the command `make new-app` or `make new-lib`. This
command will create the necessary directories and bootstrap
the application.

For example, to create a full fledged OTP application as
a local dependency:

[source,bash]
$ make new-app in=webchat

Or, the same as an OTP library:

[source,bash]
$ make new-lib in=webchat

Templates also work with local dependencies, from the root
directory of the project. You do need however to tell
Erlang.mk to create the files in the correct application:

[source,bash]
$ make new t=gen_server n=my_server in=webchat

=== Repositories with no application at the root level

It's possible to use Erlang.mk with only applications in
'$(APPS_DIR)', and nothing at the root of the repository.
Just create a folder, put the 'erlang.mk' file in it,
write a Makefile that includes it, and start creating
your applications.

Similarly, it's possible to have a repository with only
dependencies found in '$(DEPS_DIR)'. You just need to
create a Makefile and specify the dependencies you want.
This allows you to create a repository for handling the
building of releases, for example.

=== Autopatch

Erlang.mk will automatically patch all the dependencies it
fetches. It needs to do this to ensure that the dependencies
become compatible with not only Erlang.mk, but also with
the version of Erlang.mk that is currently used.

When fetching a dependency, the following operations are
performed:

* Fetch the dependency using the configured fetch method
* If it contains a 'configure.ac' or 'configure.in' file, run `autoreconf -Wall -vif -I m4`
* If it contains a 'configure' script, run it
* Run autopatch on the project

Autopatch first checks if there is any project-specific patch
enabled. There are currently three: `RABBITMQ_CLIENT_PATCH` for
the `amqp_client` dependency (before 3.6.0), `RABBITMQ_SERVER_PATCH`
for the `rabbit` dependency (before 3.6.0) and `ELIXIR_PATCH`
for the `elixir` dependency.

Otherwise, autopatch performs different operations depending
on the kind of project it finds the dependency to be.

* Rebar projects are automatically converted to use Erlang.mk
as their build tool. This essentially patches Rebar out, and
fixes and converts the project to be compatible with Erlang.mk.

* Erlang.mk projects have their 'Makefile' patched, if necessary,
to include the top-level project's Erlang.mk. This is to ensure
that functionality works across all dependencies, even if the
dependency's Erlang.mk is outdated. The patched Makefile
can be safely committed if necessary.

* Other Erlang projects get a small Erlang.mk Makefile
generated automatically.

* Projects with no source directory and no Makefile get an
empty Makefile generated, for compatibility purposes.

* Other projects with no Makefile are left untouched.

You can add additional commands to be run immediately before
or after autopatch is done by extending the target
`autopatch-$(dep)::`, for example this would remove
a module:

[source,make]
----
autopatch-ranch::
	rm -f $(DEPS_DIR)/ranch/src/ranch_proxy_header.erl
----

A common use case for this feature is to apply a PATCH
file on the dependency immediately after fetching it.
It can also be used to add compiler options, for example:

[source,make]
----
autopatch-couchbeam::
	printf "\nERLC_OPTS += -DWITH_JIFFY\n" >> $(DEPS_DIR)/couchbeam/Makefile
----

The commands will run before autopatch when the target is
defined before including 'erlang.mk', and after otherwise.

You can disable the replacing of the 'erlang.mk' file by
defining the `NO_AUTOPATCH_ERLANG_MK` variable:

[source,make]
NO_AUTOPATCH_ERLANG_MK = 1

You can also disable autopatch entirely for a few select
projects using the `NO_AUTOPATCH` variable:

[source,make]
NO_AUTOPATCH = cowboy ranch cowlib

=== Dealing with duplicate modules

When there are duplicate modules found in both applications
and their dependencies, some tasks may fail. Erlang expects
modules to be unique in general.

When the duplicates are found in dependencies, you will need
to remove one of the duplicates at fetch time. To do so, you
can add a rule similar to this to your Makefile before including
'erlang.mk':

[source,make]
----
DEPS_DIR = $(CURDIR)/deps

deps:: $(DEPS_DIR)/cowlib
	$(verbose) rm -f $(DEPS_DIR)/cowlib/src/cow_ws.erl
----

This must be done from the application that has this dependency.
Only define the `DEPS_DIR` variable if necessary.

=== Skipping deps

It is possible to temporarily skip all dependency operations.
This is done by defining the `SKIP_DEPS` variable. Use cases
include being somewhere with no connection to download them,
or perhaps a peculiar setup.

A typical usage would be:

[source,bash]
$ make SKIP_DEPS=1

When the variable is defined:

* Dependencies will not be compiled or downloaded when required
* The dependency directory '$(DEPS_DIR)' will not be removed on `make distclean`

This variable only applies to remote dependencies.