I’m working on optimising our build pipeline. I have a flow that looks like:
– compile
– dialyzer, test etc
These are executed in separate containers. The compiled files are moved from the first container to the second by saving the _build/dev directory to a workspace, which is attached to the subsequent containers.
This works well in that application files are never recompiled, however every time it always recompiles all the Erlang modules. What have I missed in order to cache these.
We are using CircleCI (which runs all builds inside docker containers). The caching options available can save/restore any files, but I suspect it resets the timestamps on restore which might be causing this issue.
When I say “compiles all the Erlang modules”, I mean the output looks like this:
Summary
mix dialyzer --halt-exit-status
===> Compiling parse_trans
===> Compiling mimerl
===> Compiling metrics
===> Compiling unicode_util_compat
===> Package unicode_util_compat-0.3.1 not found. Fetching registry updates and trying again...
===> Updating package registry...
===> Writing registry to /root/.cache/rebar3/hex/default/registry
===> Generating package index...
===> [appsignal:1.6.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.0], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.7.0-alpha.4], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.0-beta.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.3], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.7.0-alpha.3], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.7.0-alpha.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.7.0-alpha.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.5], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.4], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> [appsignal:1.6.0-alpha.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
===> Writing index to /root/.cache/rebar3/hex/default/packages.idx
===> Compiling idna
===> Compiling ranch
==> poolboy (compile)
Compiled src/poolboy_worker.erl
Compiled src/poolboy_sup.erl
Compiled src/poolboy.erl
===> Compiling hpack
==> ssl_verify_fun (compile)
Compiled src/ssl_verify_util.erl
Compiled src/ssl_verify_fingerprint.erl
Compiled src/ssl_verify_pk.erl
Compiled src/ssl_verify_hostname.erl
===> Compiling certifi
===> Compiling hackney
===> Compiling cowlib
src/cow_multipart.erl:392: Warning: call to crypto:rand_bytes/1 will fail, since it was removed in 20.0; use crypto:strong_rand_bytes/1
===> Compiling cowboy
===> Fetching rebar3_hex ({pkg,<<"rebar3_hex">>,<<"4.1.0">>})
===> Downloaded package, caching at /root/.cache/rebar3/hex/default/packages/rebar3_hex-4.1.0.tar
===> Compiling rebar3_hex
===> Compiling redbug
...dialyzer output
Not Erlang, id guess that a mix managed Erlang app wouldn’t recompile.
It’s rebar managed applications.
I think if you can manage to set up a pure Erlang project relying on rebar only which recompiles cached stuff on subsequent stages of the pipe, I’m sure the rebar people will be eager to help.
There’s multiple caches so that if for example the mix.lock changes then we can fallback to the next most recently used cache. Also the v9 is there so that when we make large changes we can cache bust everything and we can do something similar by changing the CI_CACHE_VERSION env variable (but we can change this without pushing a code update). A reason you’d need to bust the cache is when a library reads from application config during compilation (a great example is the Elixir mime type plug: https://hexdocs.pm/mime/MIME.html).
The config you posted is very close to EXACTLY what we are using :). As I stated above, this works for caching ELIXIR build files.
It does NOT work for caching the built ERLANG libs. I suspect this is because restoring a cache does not preserve the original file modification times - see mix compile docs and this circleci question.
If you watch your build logs, I think you will find that your tasks are rebuilding the erlang libs every time too.
I am trying to find a way to tell mix that it doesn’t need to recompile the erlang libs.
Related: Are there people that have experience in doing the same thing (caching Elixir and/or Erlang dependencies so that only changed things are re-compiled) for the Travis CI?
Currently the Travis builds take almost four minutes, of which running the actual test suite takes only about 15 seconds.
This Travis CI runs each of the test suites in just over a minute. The linting and dialyzer tasks are ran as jobs after the tests pass, which really speeds things up.
Just ran into a somewhat similar issue (erlang deps didn’t seem to be cached) and in my case the problem was in not caching deps folder after running mix compile. This is important since rebar3 actually stores ebins in deps, and in _build mix only creates symlinks.
So, my approach before (which had the same problem as in OP):
