How to affect number of cores elixir compilation utilizes?

Questions / Help
,
1

Hi.

We have a mid-sized project of roughly 75K lines of elixir code, spread over 500+ files.

When we compile this project on a 16 core machine, we can see in Grafana that only 4 cores are utilized at most. We would like to push this to at least 8 if possible.

The fascinating thing is that it caps out at 4 cores with no peaks beyond, making me wonder if there is some limit involved.

What controls how many cores are utilized and is there a cap, maximum, or default? I’ve been looking through compiler and mix documentation and have found nothing so far.

Thank you!

2

Elixir will use as many cores as there are scheduler threads available which defaults to the number of cores in your system.

Elixir will try to parallelize the compilation of individual files as much as possible, but any file’s compile time dependencies will have to be compiled before it which can limit the number of files that can be compiled in parallel. If you have many such dependencies in your project the compiler will not be able to use all cores.

There has been optimizations in recent Elixir releases that reduces the number of compile time dependencies so make sure you are running the latest version. @wojtekmach has also written about how to reduce the dependencies in your projects: https://dashbit.co/blog/speeding-up-re-compilation-of-elixir-projects and https://dashbit.co/blog/rewriting-imports-to-aliases-with-compilation-tracers.

1 Like
3

Hi, @ericmj.
The machine is listed as having 16 virtual cores, being a cloud instance. Yet elixir only utilizes only 4 at any time.
I have hundreds of files, I doubt that there is no combination of 5 that can’t be compiled independently.

elixir release was 1.10.4. We can’t go forward beyond that right now for reasons having to do with the organization maintaining the infrastructure.

What system call or variables does elixir evaluate to determine how many cores to use for compilation? I could query those to learn more about why this happens.

4

If you run iex you will get an output such as:

Erlang/OTP 23 [erts-11.0.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe]

Interactive Elixir (1.12.0-dev) - press Ctrl+C to exit (type h() ENTER for help)

The number in [smp:12:12] will say how many scheduler threads are being used.

1 Like
5

$ /opt/elixir/x86_64/1.10.4/bin/iex
Erlang/OTP 22 [erts-10.7.2.1] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1]

Seems to be recognized as having 16 cores alright.

6

In that case the compiler will use 16 cores unless restricted by compile time dependencies reducing concurrency.

7

So, in principle I could add a dozen or so .ex files with no external dependencies, all in the same folder. Would I expect the core utilization to go up, or is it less deterministic than that?

If it helps the experiment, I could generate some code to make sure compilation takes a while.

8

Another good test would be to try compiling a different project

9

Also a good idea! Any good candidates that parallelize well?

If none come to mind, I would just generate modules with a thousand functions heads each, like def add1(1), do: 2 for a long range of scalars.

10

Yes, core utilization should go up if you have enough files and they are large enough, at least for part of the compilation.

11

totally understandable - but do give it a go on latest stable elixir and perhaps even master, to see/check if there are any gains there, potentially the entire solution is there… (and yes deploying the “fix” will be a future thing then…)

12

Yes, maybe I can.

While I don’t have control over the CI machine, maybe I could change the job script so that pulls in the elixir tarball, builds it, and then let it use that one instead of the one we have deployed. Maybe. :wink:

1 Like