jeroenvisser101
I noticed that while compiling an application, ParallelCompiler will compile multiple modules in parallel, as long as the dependencies between them allow this, but that compiling mix dependencies does not. It compiles each package one-by-one. While some packages really take advantage of all cores, others do not, and I wonder if there are any constraints that prevent Elixir from monitoring parallelism within a package and if it’s making proper use of all cores.
I imagine that compiling all dependencies could be a multi-step process, first scanning and sorting by dependency on others, and then, similar to GenStage and Flow, dispatch modules to be compiled as events to System.schedulers_online, making sure that each consumer is compiling at least one module.
I’m looking to hear other people’s thoughts on this, as well as seeing how your dependency compilation times and cpu usage compare (time mix deps.compile --force), and if they are making optimal use of all your cores.
My result:
MacBook Pro 2019, 2.4GHz 8 core i9: mix deps.compile --force 231.52s user 25.97s system 317% cpu 1:21.12 total
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Most Liked- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
hauleth
You cannot monitor NIFs or other “external” build tools. So this could happen that you would run in parallel 2 packages that would try to compile some NIF with
make -j8on machine with 8 cores and it would in the end take longer due to constant scope swapping between the two compilation processes.tristan
I’ve wanted to introduce this in rebar3 from the beginning but never got around to it. We have a big rework of the compiler in the works and hope it can be introduced then Compiler Refactor Extravaganza · Issue #2200 · erlang/rebar3 · GitHub
Would be cool to see it done in Mix as well.
jeroenvisser101
It could even be an opt-in feature, allowing you to execute
mix deps.compile --parallel, and as we find cases that cause problems, we can tackle them. After some time, it could become the default with a flag to disable.Looks very promising!
Or
compile_parallel: falseto disable, or as I mentioned, allow package authors to specify if it supports being compiled in parallel with others. Additionally, likeextra_applications, you could specify a allow/block-list to override parallel compiler compatibility.