setop
Hi all,
I’m maintaining a faily large Elixir project (100 direct dependencies, 177MB deps, 554 source files, 56MB _build/dev).
I release it as a docker image.
My issue is that docker build is very long as it has to download dependencies, build dependencies, build sources, each time.
I’m looking for tips to accelerate this step, maybe using cache, since dependencies do not vary much and sources are largely the same from one build to the next.
I already reordered instruction to avoid docker cache invalidation if nothing changes (as explained here). But if only a tiny piece is change (eg: new version of a deps), the whole layer is invalidated.
Any idea ?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 6 of 6 Posts
dimitarvp
Just to narrow the list of suspects first: you’re using a multi-stage Dockerfile, right? And
mix deps.getandmix deps.compileare both separate commands, not mixed with others on the same line, correct?Oh, and this: are you cross-compiling for another CPU architecture?
setop
Yes
Yes
Nope
dimitarvp
Nothing quick jumps to mind then. I suggest you post your Dockerfile so others can chime in.
jozuas
If you have the time to invest into learning Nix, you can build Docker containers with Nix.
EDIT: Updated to a higher quality video
setop
Thanks ! Look promising. Plus I always wanted to have a try at grasping nix (it is on my machine but I don’t use it much yet). I’ll make some experiment and post result here.
PR : for those who want to spare time, the meat of the talk starts at 25:07.
al2o3cr
A general approach in situations like this is to split the work that’s done in the layer into pieces that can be individually invalidated.
When I saw a similar issue with a Rails project (gems with native extensions take a long time to install), the solution was to have a separate step that only installed those gems before the main
bundle install. That was simpler to implement because the gems were going to the system location (versus building in a Mix project).Instead of a single
mix deps.getandmix deps.compile, you could have a “mix-foundation.exs” (withlockfileset tomix-foundation.lock) then useMIX_EXS=mix-foundation.exs mix deps.getetcThat should produce a rarely-invalidated layer that pre-populates the
_builddirectory and speeds up the plainmix deps.compilelater.One thing to watch out for: if the versions in the main file drift away from the ones in the foundation, nothing will fail but build times will go up when the correct versions have to also be installed. (the Ruby version had been in place for years and had this exact drift)