Oliver

Oliver

How to affect number of cores elixir compilation utilizes?

Hi.

We have a mid-sized project of roughly 75K lines of elixir code, spread over 500+ files.

When we compile this project on a 16 core machine, we can see in Grafana that only 4 cores are utilized at most. We would like to push this to at least 8 if possible.

The fascinating thing is that it caps out at 4 cores with no peaks beyond, making me wonder if there is some limit involved.

What controls how many cores are utilized and is there a cap, maximum, or default? I’ve been looking through compiler and mix documentation and have found nothing so far.

Thank you!

Most Liked

Oliver

Oliver

Well, we eventually upgraded to 1.14.4 (OTP25), and I remember expecting it to improve because dependency analysis is said to be improved, but the load average stays at 4 for 16 virtual cores.

Ironically, I found this thread when trying to analyze the same problem four years later. :stuck_out_tongue: I just didn’t check back then when we upgraded, but I see load averages remain where they were, I guess.

DianaOlympos

DianaOlympos

Something to note. Average core load of 4 does not mean only 4 cores are used

It could be that 16 are used but they spend most of their time on IO or idling or blocked by a lock somewhere. This is a not too rare thing to happen.

CPU load is a pretty flawed metric these days to evaluate how much you saturate the number of cores.

See CPU Utilization is Wrong and Linux Load Averages: Solving the Mystery as starting points

ericmj

ericmj

Elixir Core Team

Elixir will use as many cores as there are scheduler threads available which defaults to the number of cores in your system.

Elixir will try to parallelize the compilation of individual files as much as possible, but any file’s compile time dependencies will have to be compiled before it which can limit the number of files that can be compiled in parallel. If you have many such dependencies in your project the compiler will not be able to use all cores.

There has been optimizations in recent Elixir releases that reduces the number of compile time dependencies so make sure you are running the latest version. @wojtekmach has also written about how to reduce the dependencies in your projects: https://dashbit.co/blog/speeding-up-re-compilation-of-elixir-projects and Rewriting imports to aliases with compilation tracers - Dashbit Blog.

Where Next?

Popular in Questions Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

We're in Beta

About us Mission Statement