setop
How to accelerate docker build?
Hi all,
I’m maintaining a faily large Elixir project (100 direct dependencies, 177MB deps, 554 source files, 56MB _build/dev).
I release it as a docker image.
My issue is that docker build is very long as it has to download dependencies, build dependencies, build sources, each time.
I’m looking for tips to accelerate this step, maybe using cache, since dependencies do not vary much and sources are largely the same from one build to the next.
I already reordered instruction to avoid docker cache invalidation if nothing changes (as explained here). But if only a tiny piece is change (eg: new version of a deps), the whole layer is invalidated.
Any idea ?
Most Liked
al2o3cr
A general approach in situations like this is to split the work that’s done in the layer into pieces that can be individually invalidated.
When I saw a similar issue with a Rails project (gems with native extensions take a long time to install), the solution was to have a separate step that only installed those gems before the main bundle install. That was simpler to implement because the gems were going to the system location (versus building in a Mix project).
Instead of a single mix deps.get and mix deps.compile, you could have a “mix-foundation.exs” (with lockfile set to mix-foundation.lock) then use MIX_EXS=mix-foundation.exs mix deps.get etc
That should produce a rarely-invalidated layer that pre-populates the _build directory and speeds up the plain mix deps.compile later.
One thing to watch out for: if the versions in the main file drift away from the ones in the foundation, nothing will fail but build times will go up when the correct versions have to also be installed. (the Ruby version had been in place for years and had this exact drift)
dimitarvp
Just to narrow the list of suspects first: you’re using a multi-stage Dockerfile, right? And mix deps.get and mix deps.compile are both separate commands, not mixed with others on the same line, correct?
Oh, and this: are you cross-compiling for another CPU architecture?
jozuas
If you have the time to invest into learning Nix, you can build Docker containers with Nix.
EDIT: Updated to a higher quality video
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #javascript
- #code-sync
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"