Parallel compiler and mix dependencies

You cannot monitor NIFs or other “external” build tools. So this could happen that you would run in parallel 2 packages that would try to compile some NIF with make -j8 on machine with 8 cores and it would in the end take longer due to constant scope swapping between the two compilation processes.

Thanks, good point!

Maybe not monitor, or run external build tools in parallel, but that shouldn’t (have to) mean that Elixir/erlang-only packages couldn’t run in parallel?

I think it may make scheduling more complicated, and I had not thought about this aspect, but I think even if only parts are parallelized that could already be improving performance. Acquiring a ‘lock’ or exclusivity on system resources for external build tools, possibly configurable by the package author, might prevent it from having to switch constantly.

The problem is that you have no way to know whether it can or cannot be run in parallel. In theory there could be added an additional option in the compilers to allow them to compile applications independently, but I think it would be a little too much work for little gain, especially as files within each project are compiled independently whenever that is possible.

If I understand correctly, using an compiler other than Elixir requires additional configuration for use with mix, which could be used to filter dependencies that can be compiled solely with the Elixir compiler, and by using LexicalTracker on those projects, both internal and cross-project dependencies could be tracked and resolved when those become available?

Even if a compiler would have to require a global lock (e.g. be the only compiler running), I think it would still take better use of available compute resources if it was able to parallelize partially.

Thing is—I’m not sure how long average projects take to compile their dependencies. I can imagine speed improvements in compiling dependencies being irrelevant to many as their cache often is enough to prevent subsequent recompilation. The reason for asking is only because I noticed that while updating Elixir and Erlang, compilation used all cores, but recompiling the project barely used half of available resources and I wondered why

Regardless if the gains will be big or small, I’d love to see more effort in further parallelising building your Elixir project, dependencies included. I am also aware of the problems @hauleth mentioned but it shouldn’t be hard to just always compile packages that contain NIFs serially? IMO it’s pretty trivial to detect.

From then on, all pure Erlang/Elixir dependencies and the Elixir project itself should be a fair game for maximally parallelised compilation.

I’d like to see all my 10/20 CPU cores maxed while building!

Oh really? How? There can be a compiler defined for each of different languages. I could even write assembler in Elixir (because why f…reaking not). How would you detect whether I am not doing something weird? Or that I am generating some magical files somewhere (even from Elixir itself) that are used somewhere else. I am not even sure how it would work with setting application environment variables during compilation. Due to expressiveness of Elixir the parallel compilation of different projects at once can go south enormously fast.

If I remember well, there are certain pieces of code that you have to write (in C) if you like to actually interface with a NIF. That is not hard to detect.

You are right there are code-generating cases though, I haven’t considered them. I guess for those cases I’d be fine if 4-5 NIF-relying packages compile a bit slower because they compete too much.

My point is, I get this little annoying feeling when I see projects with 400+ files compiling for 12 secs on a beefy workstation and the CPU cores never pass 20-30% load each. You know what I mean. I wonder if there are low-hanging fruits to pick there.

Yeah, this is partially true, however (again) due to the dynamic nature of Elixir, it is hard to detect is something will be done unexpectedly.

Yes, under one circumstance. However if the native code is built, but it is not a NIF (but C Node, Port, Port Driver, etc.) then this became less obvious.

I’ve wanted to introduce this in rebar3 from the beginning but never got around to it. We have a big rework of the compiler in the works and hope it can be introduced then Compiler Refactor Extravaganza · Issue #2200 · erlang/rebar3 · GitHub

Would be cool to see it done in Mix as well.

True true. I didn’t account for all possibilities. A manual switch in the package itself (has_nif: true for example?) could also help. Can’t imagine it being that hard to introduce and default to false if not present. As mentioned above, might make several NIF packages fight for your CPU cores but eh, is that a doomsday scenario?

Pretty cool!