After running mix clean and then say mix test I see all my files and modules compile one by one. During the process I have plenty of CPU underutilized and I ask myself why the compiler can’t utilize more cores on my machine?
PS: I’m not a compiler geek so I’m not looking for a deep answer, just kind of curious.
Compiling one of my bigger projects used all 6 of my cores pretty well? They should only compile one by one if the dependency structure is really bad?
Is it possible you are conflating compiler parallelism and test parallelism? Do you see a difference is you just do mix compile —force? At the point you should see many cores being used unless, as @OvermindDL1 says, your dependency graph prevents it.
My comment is about compiler parallelism specifically. Without doing any real measurement the -force options does seem to push a little harder. 
My curiosity was from my iOS development, where I do see Xcode pegging all my cores over the course of a 100 second build – but when I do Elixir I never see the CPUs pegged.
Thanks for the info.
Do you have a lot of front-end assets being processed in a pipeline? (Webpack, brunch, etc…) If so, that would account for a long wait on node.js which will only use a bit more than 1 core.
[EDIT] I process my front end separately, and it’s super slow compared to Elixir. No front end assets in my Elixir builds, and it tends to peg most cores most of the time.
Yes, Elixir compiles files in parallel for a long time! 
It is one of the first articles published on the official website (for those who are interested in the technical aspects).
How large is your project? It may be that your project is small enough (and compilation is fast enough) that we actually don’t have enough time to warm up all of your CPUs. For example, when compiling hexpm, my quad-core maxes at 91% core usage and it still finishes in 5.5 seconds. When you say you see all your files and modules one by one, how are you visualizing that?
It may be that your project is small enough (and compilation is fast enough) that we actually don’t have enough time to warm up all of your CPUs.
That could be true. These are not large projects.
When you say you see all your files and modules one by one, how are you visualizing that?
I see it compiling after I issue a test command via mix or by launching the Phoenix server.
mix test
===> Compiling parse_trans
==> gen_state_machine
Compiling 3 files (.ex)
Generated gen_state_machine app
==> pane
Compiling 3 files (.ex)
warning: Enum.chunk/4 is deprecated. Use Enum.chunk_every/4 instead
lib/page.ex:44
Generated pane app
==> scribe
Compiling 12 files (.ex)
Generated scribe app
No more discussion needed for my part. Thanks for all the info.
Only thing --force does is recompile everything, even unchanged files. 
Elixir also compiles a LOT faster than most optimized compiled languages like swift/objc(/c/c++/rust/etc…) so it could just be running fast enough to not catch it.
Now the mix build system will build ‘dependencies’ one at a time, rather than in parallel as the graph allows, that could probably be fixed, but its generally not an issue either as those don’t recompile often, but ‘inside’ each dependency is still compiled in parallel too, which makes parallel dependency building even less important.
Yeah front-end assets is obviously not elixir, so that is all based on the node system, I.E. inherently very single-core.
On this, things like Compiling 12 files (.ex) means it is compiling 12 files from the dependency scribe in parallel (as their dependency graph allows).
© Copyright Elixir Forum | Terms | Privacy & Cookies
