Oliver
Hi.
We have a mid-sized project of roughly 75K lines of elixir code, spread over 500+ files.
When we compile this project on a 16 core machine, we can see in Grafana that only 4 cores are utilized at most. We would like to push this to at least 8 if possible.
The fascinating thing is that it caps out at 4 cores with no peaks beyond, making me wonder if there is some limit involved.
What controls how many cores are utilized and is there a cap, maximum, or default? I’ve been looking through compiler and mix documentation and have found nothing so far.
Thank you!
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Elixir, Erlang & messaging conferences from Code Sync. Next up: ElixirConf US (10–11 Sep) & Code BEAM Europe (21–22 Oct), Haarlem + online.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
ericmj
Elixir will use as many cores as there are scheduler threads available which defaults to the number of cores in your system.
Elixir will try to parallelize the compilation of individual files as much as possible, but any file’s compile time dependencies will have to be compiled before it which can limit the number of files that can be compiled in parallel. If you have many such dependencies in your project the compiler will not be able to use all cores.
There has been optimizations in recent Elixir releases that reduces the number of compile time dependencies so make sure you are running the latest version. @wojtekmach has also written about how to reduce the dependencies in your projects: https://dashbit.co/blog/speeding-up-re-compilation-of-elixir-projects and Rewriting imports to aliases with compilation tracers - Dashbit Blog.
Oliver
Hi, @ericmj.
The machine is listed as having 16 virtual cores, being a cloud instance. Yet elixir only utilizes only 4 at any time.
I have hundreds of files, I doubt that there is no combination of 5 that can’t be compiled independently.
elixir release was 1.10.4. We can’t go forward beyond that right now for reasons having to do with the organization maintaining the infrastructure.
What system call or variables does elixir evaluate to determine how many cores to use for compilation? I could query those to learn more about why this happens.
ericmj
If you run
iexyou will get an output such as:The number in
[smp:12:12]will say how many scheduler threads are being used.Oliver
Seems to be recognized as having 16 cores alright.
ericmj
In that case the compiler will use 16 cores unless restricted by compile time dependencies reducing concurrency.
Oliver
So, in principle I could add a dozen or so .ex files with no external dependencies, all in the same folder. Would I expect the core utilization to go up, or is it less deterministic than that?
If it helps the experiment, I could generate some code to make sure compilation takes a while.
axelson
Another good test would be to try compiling a different project
Oliver
Also a good idea! Any good candidates that parallelize well?
If none come to mind, I would just generate modules with a thousand functions heads each, like
def add1(1), do: 2for a long range of scalars.ericmj
Yes, core utilization should go up if you have enough files and they are large enough, at least for part of the compilation.
outlog
totally understandable - but do give it a go on latest stable elixir and perhaps even master, to see/check if there are any gains there, potentially the entire solution is there.. (and yes deploying the “fix” will be a future thing then..)