jeroenvisser101
Parallel compiler and mix dependencies
I noticed that while compiling an application, ParallelCompiler will compile multiple modules in parallel, as long as the dependencies between them allow this, but that compiling mix dependencies does not. It compiles each package one-by-one. While some packages really take advantage of all cores, others do not, and I wonder if there are any constraints that prevent Elixir from monitoring parallelism within a package and if it’s making proper use of all cores.
I imagine that compiling all dependencies could be a multi-step process, first scanning and sorting by dependency on others, and then, similar to GenStage and Flow, dispatch modules to be compiled as events to System.schedulers_online, making sure that each consumer is compiling at least one module.
I’m looking to hear other people’s thoughts on this, as well as seeing how your dependency compilation times and cpu usage compare (time mix deps.compile --force), and if they are making optimal use of all your cores.
My result:
MacBook Pro 2019, 2.4GHz 8 core i9: mix deps.compile --force 231.52s user 25.97s system 317% cpu 1:21.12 total
Most Liked
hauleth
You cannot monitor NIFs or other “external” build tools. So this could happen that you would run in parallel 2 packages that would try to compile some NIF with make -j8 on machine with 8 cores and it would in the end take longer due to constant scope swapping between the two compilation processes.
tristan
I’ve wanted to introduce this in rebar3 from the beginning but never got around to it. We have a big rework of the compiler in the works and hope it can be introduced then Compiler Refactor Extravaganza · Issue #2200 · erlang/rebar3 · GitHub
Would be cool to see it done in Mix as well.
jeroenvisser101
It could even be an opt-in feature, allowing you to execute mix deps.compile --parallel, and as we find cases that cause problems, we can tackle them. After some time, it could become the default with a flag to disable.
Looks very promising!
Or compile_parallel: false to disable, or as I mentioned, allow package authors to specify if it supports being compiled in parallel with others. Additionally, like extra_applications, you could specify a allow/block-list to override parallel compiler compatibility.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"