Why Elixir dependencies compilation artifacts end up in deps?

Questions / Help
1

I am trying to understand Elixir better following problems with caching deps directory in my GitLab CI pipeline.

  • mix deps.get fetches dependencies and puts them in the deps directory
  • mix deps.compile compiles the dependencies by creating files in both the _build directory and the deps directory (deps/<DEPENDENCY>/_build and deps/<DEPENDENCY>/ebin/*.beam)

Example:

$ rm -r deps/

$ rm -r _build/

$ mix deps.get 
Resolving Hex dependencies...
Resolution completed in 0.476s
Unchanged:
  […]
  ranch 1.8.0
  […]
* […]
* Getting ranch (Hex package)
* […]
You have added/upgraded packages you could sponsor, run `mix hex.sponsor` to learn more

$ tree -a deps/ranch/
deps/ranch/
├── .fetch
├── .hex
├── LICENSE
├── Makefile
├── README.asciidoc
├── ebin
│   └── ranch.app
├── erlang.mk
├── hex_metadata.config
└── src
    ├── ranch.erl
    ├── ranch_acceptor.erl
    ├── ranch_acceptors_sup.erl
    ├── ranch_app.erl
    ├── ranch_conns_sup.erl
    ├── ranch_crc32c.erl
    ├── ranch_listener_sup.erl
    ├── ranch_protocol.erl
    ├── ranch_proxy_header.erl
    ├── ranch_server.erl
    ├── ranch_ssl.erl
    ├── ranch_sup.erl
    ├── ranch_tcp.erl
    └── ranch_transport.erl

2 directories, 22 files

$ mix deps.compile ranch
===> Analyzing applications...
===> Compiling ranch

$ tree -a deps/ranch/
deps/ranch/
├── .fetch
├── .hex
├── LICENSE
├── Makefile
├── README.asciidoc
├── _build
│   └── prod
│       └── lib
│           └── .rebar3
│               └── rebar_compiler_erl
│                   └── source.dag
├── ebin
│   ├── ranch.app
│   ├── ranch.beam
│   ├── ranch_acceptor.beam
│   ├── ranch_acceptors_sup.beam
│   ├── ranch_app.beam
│   ├── ranch_conns_sup.beam
│   ├── ranch_crc32c.beam
│   ├── ranch_listener_sup.beam
│   ├── ranch_protocol.beam
│   ├── ranch_proxy_header.beam
│   ├── ranch_server.beam
│   ├── ranch_ssl.beam
│   ├── ranch_sup.beam
│   ├── ranch_tcp.beam
│   └── ranch_transport.beam
├── erlang.mk
├── hex_metadata.config
└── src
    ├── ranch.erl
    ├── ranch_acceptor.erl
    ├── ranch_acceptors_sup.erl
    ├── ranch_app.erl
    ├── ranch_conns_sup.erl
    ├── ranch_crc32c.erl
    ├── ranch_listener_sup.erl
    ├── ranch_protocol.erl
    ├── ranch_proxy_header.erl
    ├── ranch_server.erl
    ├── ranch_ssl.erl
    ├── ranch_sup.erl
    ├── ranch_tcp.erl
    └── ranch_transport.erl

7 directories, 37 files

$ tree -a _build/dev/lib/ranch/
_build/dev/lib/ranch/
├── .mix
│   └── compile.fetch
├── ebin -> ../../../../deps/ranch/ebin
└── mix.rebar.config

2 directories, 2 files

Why is not mix deps.compile limiting itself to create files in the _build directory? After all the _build directory is named so for a reason.

I am sure there is a good reason for that. I am just trying to understand the rational.

2

My understanding is that the reason mix deps.compile creates files in both the deps and _build directories is to maintain compatibility with the Erlang ecosystem and to handle Elixir-specific build artifacts.

The symlink is used to make the compiled BEAM files from the deps directory available to the Elixir project for testing and development.

3

Thank you for your comment, do you have any link to official documentation that confirms this?

4

I would be surprised if there was any; the structure and contents of the deps and _build directories are meant to be “private” implementation details of mix.

5

Generally all compilation output should go in _build. Although sometimes that take a bit of work to setup. I remember seeing some PRs to projects that use elixir_make to switch those projects to put their compilation output and intermediate files in _build

6

Thank you everyone for your help!

For reference : I add the link to the post What directory structure and files does mix create? - #4 by joeerl which also tried to understand the directory structure.

The idea that the content of deps and _build directories are meant to be “private” implementation details is also problematic to me. I use to cache deps and add _build as artifact in my GitLab CI pipeline which uses several concurrent jobs. It took me extremely long time to understand that the reason my GitLab CI pipeline sometime failed and sometime passed was partly due to deps containing build artifacts.
So in practice, it is not an implementation detail because I needed that knowledge (deps containing build artifacts) to achieve my goal.