Oliver
How to affect number of cores elixir compilation utilizes?
Hi.
We have a mid-sized project of roughly 75K lines of elixir code, spread over 500+ files.
When we compile this project on a 16 core machine, we can see in Grafana that only 4 cores are utilized at most. We would like to push this to at least 8 if possible.
The fascinating thing is that it caps out at 4 cores with no peaks beyond, making me wonder if there is some limit involved.
What controls how many cores are utilized and is there a cap, maximum, or default? I’ve been looking through compiler and mix documentation and have found nothing so far.
Thank you!
Most Liked
Oliver
Well, we eventually upgraded to 1.14.4 (OTP25), and I remember expecting it to improve because dependency analysis is said to be improved, but the load average stays at 4 for 16 virtual cores.
Ironically, I found this thread when trying to analyze the same problem four years later.
I just didn’t check back then when we upgraded, but I see load averages remain where they were, I guess.
DianaOlympos
Something to note. Average core load of 4 does not mean only 4 cores are used
It could be that 16 are used but they spend most of their time on IO or idling or blocked by a lock somewhere. This is a not too rare thing to happen.
CPU load is a pretty flawed metric these days to evaluate how much you saturate the number of cores.
See CPU Utilization is Wrong and Linux Load Averages: Solving the Mystery as starting points
ericmj
Elixir will use as many cores as there are scheduler threads available which defaults to the number of cores in your system.
Elixir will try to parallelize the compilation of individual files as much as possible, but any file’s compile time dependencies will have to be compiled before it which can limit the number of files that can be compiled in parallel. If you have many such dependencies in your project the compiler will not be able to use all cores.
There has been optimizations in recent Elixir releases that reduces the number of compile time dependencies so make sure you are running the latest version. @wojtekmach has also written about how to reduce the dependencies in your projects: https://dashbit.co/blog/speeding-up-re-compilation-of-elixir-projects and Rewriting imports to aliases with compilation tracers - Dashbit Blog.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









