japhib

Why are compile-time dependencies between modules transitive?

I use Elixir at work, on a pretty massive monolithic backend project – it’s about 750k lines of Elixir code, spread across a few dozen apps, all within one big umbrella app. It seems like this is fairly uncommon, as most Elixir apps I hear about online or in various forums are much smaller. Even the Elixir codebase itself is less than half as big as that.

One of the main gripes I have of working with Elixir is the slow compile times. Of course, compiling a large project is expected to take a long time. However, even when I make the smallest changes imaginable, Elixir decides to recompile many files, resulting in a frustratingly slow dev cycle. I’m wondering if anything can be done about this.

The concrete example is that when I add a single IO.inspect() in a module deep in the dependency tree, it requires many other files to be recompiled before it can proceed. Here’s my terminal output from when I did that recently (keep in mind this is only adding a single IO.inspect() call in 1 file:)

(Note: mt in my terminal command is just an alias for mix test)

See the timestamps on the right – re-compilation took over 1 whole minute after making that tiny change! The reason being, of course, that Elixir decided to recompile nearly 100 files across 12 other apps in the umbrella app.

I’d love to share the code to make the example clearer, but of course, that code belongs to my company and is not open source. So I’ll have to resort to hypothetical examples.

Now, I understand that if file A is depended on by file B, then if file A changes, file B should be recompiled as well. What I don’t understand, and the purpose of this post is to ask about, is why files that transitively depend on a changed file must also be recompiled?

Said another way:

Let’s say you have 5 files in a project: A, B, C, D, and E. Each file depends on only one other file, but it’s in a line, like so: A ← B ← C ← D ← E – so E depends only on D, D depends only on C, C depends only on B, B depends only on A, etc.
Change file A. For example, add an IO.inspect or something simple.
Run mix compile. Elixir will now recompile, not just the changed file (A) and the one that depends on it (B), but every file that depends on another file that depends on A – in other words, the entire project: files A, B, C, D, and E.

Why is this? Can anything be done about it?

I haven’t contributed to the Elixir codebase before, but I’d love to learn how. I feel confident that any time spent improving this situation would save a lot of time for devs working at my company, as well as any other devs working on a large codebase like this.

Thank you for anyone who took the time to read this post! I look forward to any responses.

15 comments

#compiling #language

41 2650 15

2023-04-04 22:42:01 UTC

Most Liked

josevalim

Creator of Elixir

Besides the excellent replies above, there are likely places we can optimize Mix or the compiler. It should not take 1 minute to compile 100 files, so I am assuming the slowdown is elsewhere and not in the compiler per-se. Can you please try doing the same IO.inspect change as before and then running:

MIX_DEBUG=1 mix compile --profile=time

and putting the output in a gist? I am particularly interested in the different between these times:

[profile] Finished compilation cycle of 2 modules in 5ms
[profile] Finished group pass check of 2 modules in 0ms
<- Ran mix compile.elixir in 21ms

My theory is that there is a large overhead on mix compile.elixir compared to the actual compiler. Looking at the code, I can think of some ideas for improvements, so I will try them out now.

Btw, if you are interested, please ask your employer if they would be willing to share it with me. If necessary, we could sign a terms of service through Dashbit so we are covered by confidentiality agreements.

Post #4

axelson

Scenic Core Team

mix xref graph is the main tool for tracking down compile-time dependencies.

If you haven’t seen it https://medium.com/multiverse-tech/how-to-speed-up-your-elixir-compile-times-part-2-test-your-understanding-f6ff3de5eb5d is a great article that really walks you through what causes a compilation dependencies, including the surprising/tricky bit about transitivity.

Another tool you could try is DepViz, which I wrote when I encountered the same pain that you’re encountering: DepViz - A visual tool to understand inter-file dependencies - #5 by axelson

DepViz will give you a view similar to what you’re asking for. Imagine you’re trying to figure out why create_connection.ex is recompiled, you hover over that file:

From that you can see that there’s 16 files that will cause create_connection.ex to be recompiled. Of those, 3 are direct dependencies, but 15 of them are transitive dependencies via the compile-time dependency on command.ex
(side-note: something is not quite right in the math there! Probably the other two top-level files are being double-counted because there’s a comptile-time dependency loop)

If you want to dig further into how a specific file is reached via command.ex then first click on create_connection.ex, then click on a file (e.g. write_local_name.ex):

From that we can now see that there is the dependency chain:

create_connection.ex
command.ex
write_local_name.ex

Where create_connection.ex depends on command.ex at compile-time, and command.ex depends on write_local_name.ex (at compile-time as indicated by the red dotted lines, although a run-time dependency between these two files would also create a transitive compile-time dependency for create_connection.ex).

I hope this helps you track down your compile-time dependencies!

Post #16