Why are compile-time dependencies between modules transitive?

[japhib]

I use Elixir at work, on a pretty massive monolithic backend project – it’s about 750k lines of Elixir code, spread across a few dozen apps, all within one big umbrella app. It seems like this is fairly uncommon, as most Elixir apps I hear about online or in various forums are much smaller. Even the Elixir codebase itself is less than half as big as that.

One of the main gripes I have of working with Elixir is the slow compile times. Of course, compiling a large project is expected to take a long time. However, even when I make the smallest changes imaginable, Elixir decides to recompile many files, resulting in a frustratingly slow dev cycle. I’m wondering if anything can be done about this.

The concrete example is that when I add a single IO.inspect() in a module deep in the dependency tree, it requires many other files to be recompiled before it can proceed. Here’s my terminal output from when I did that recently (keep in mind this is only adding a single IO.inspect() call in 1 file:)

(Note: mt in my terminal command is just an alias for mix test)

See the timestamps on the right – re-compilation took over 1 whole minute after making that tiny change! The reason being, of course, that Elixir decided to recompile nearly 100 files across 12 other apps in the umbrella app.

I’d love to share the code to make the example clearer, but of course, that code belongs to my company and is not open source. So I’ll have to resort to hypothetical examples.

Now, I understand that if file A is depended on by file B, then if file A changes, file B should be recompiled as well. What I don’t understand, and the purpose of this post is to ask about, is why files that transitively depend on a changed file must also be recompiled?

Said another way:

• Let’s say you have 5 files in a project: A, B, C, D, and E. Each file depends on only one other file, but it’s in a line, like so: A ← B ← C ← D ← E – so E depends only on D, D depends only on C, C depends only on B, B depends only on A, etc.
• Change file A. For example, add an IO.inspect or something simple.
• Run mix compile. Elixir will now recompile, not just the changed file (A) and the one that depends on it (B), but every file that depends on another file that depends on A – in other words, the entire project: files A, B, C, D, and E.

Why is this? Can anything be done about it?

I haven’t contributed to the Elixir codebase before, but I’d love to learn how. I feel confident that any time spent improving this situation would save a lot of time for devs working at my company, as well as any other devs working on a large codebase like this.

Thank you for anyone who took the time to read this post! I look forward to any responses.

[al2o3cr]

A chain like A ← B ← C can happen very easily:

• A defines a macro
• B uses the macro to generate a function foo/2
• C calls B.foo(x, y)

Changing the definition in A to instead produce a foo/3 and recompiling should fail, but that will only happen if C is recompiled.

There’s a lot more information in the docs for mix xref, which you’ll need to get familiar with to to figure out what’s going on in your application.

Also make sure you’re on the latest Elixir version if at all possible - the core team has done a lot of work behind-the-scenes to try to reduce these kind of headaches in the last couple releases.

[mindok]

There are a couple of good posts explaining some of the issues and how to analyse them.

e.g. Understanding and fixing recompilation in Elixir projects, Implicit compile-time dependencies, How to speed up your Elixir compile times (part 1) — understanding Elixir compilation | by Tim Gent | multiverse-tech | Medium

[josevalim]

Besides the excellent replies above, there are likely places we can optimize Mix or the compiler. It should not take 1 minute to compile 100 files, so I am assuming the slowdown is elsewhere and not in the compiler per-se. Can you please try doing the same IO.inspect change as before and then running:

MIX_DEBUG=1 mix compile --profile=time

and putting the output in a gist? I am particularly interested in the different between these times:

[profile] Finished compilation cycle of 2 modules in 5ms
[profile] Finished group pass check of 2 modules in 0ms
<- Ran mix compile.elixir in 21ms

My theory is that there is a large overhead on mix compile.elixir compared to the actual compiler. Looking at the code, I can think of some ideas for improvements, so I will try them out now.

---

Btw, if you are interested, please ask your employer if they would be willing to share it with me. If necessary, we could sign a terms of service through Dashbit so we are covered by confidentiality agreements.

[michallepicki]

How about this scenario? Given these modules:

$ cat lib/a.ex 
defmodule A do
  def foo(), do: "hey"
end

$ cat lib/b.ex
defmodule B do
  def bar() do
    A.foo()
  end

  defmacro __using__(_opts) do
    []
  end
end

$ cat lib/c.ex
defmodule C do
  use B
end

$ cat lib/d.ex
defmodule D do
  use B
end

I’m wondering if the recompilation cascade could be “stopped” if some module in a dependency chain didn’t change in a meaningful way (its bytecode stayed the same). Currently if I modify A then both dependencies of B get recompiled, even though B didn’t really change.

$ # modify lib/a.ex

$ mix compile --verbose
Compiling 3 files (.ex)
Compiled lib/a.ex
Compiled lib/d.ex
Compiled lib/c.ex

[LostKobrakai]

@michallepicki

I’m wondering if the recompilation cascade could be “stopped” if some module in a dependency chain didn’t change in a meaningful way (its bytecode stayed the same). Currently if I modify A then both dependencies of B get recompiled, even though B didn’t really change.

B doesn’t need to change for the change in A to have effects on C and D. Call A.foo within B.__using__() and you have C and D actually depending on A.

[michallepicki]

Right. So in this case the compiler would have to be much smarter to see that this runtime dependency shouldn’t be transitive to compile dependencies because it’s not used in an exposed macro, tricky…

But then if I add one more level of spaghetti macros:

$ cat lib/aa.ex
defmodule AA do
  defmacro __using__(_) do
    []
  end

  def baz(), do: "hey"
end

$ cat lib/a.ex 
defmodule A do
  use AA
  def foo(), do: "hey"
end

and I modify AA.baz to return hey2, the bytecode of A won’t change, but C and D will get recompiled:

$ mix compile --verbose
Compiling 4 files (.ex)
Compiled lib/c.ex
Compiled lib/d.ex
Compiled lib/aa.ex
Compiled lib/a.ex

[japhib]

Thank you for the information! I will be sure to read the resources provided.

@al2o3cr

A chain like A ← B ← C can happen very easily:

• A defines a macro
• B uses the macro to generate a function foo/2
• C calls B.foo(x, y)

Changing the definition in A to instead produce a foo/3 and recompiling should fail, but that will only happen if C is recompiled.

That makes sense. It seems like this particular case would be solved in a fairly straightforward way – when recompiling a module, if it’s a runtime dependency (C calls B.foo(x, y)), keep track of the exported functions & their arities. If the exported functions/arities don’t change when recompiling, then there’s no need to recompile modules with runtime-only dependencies on that module.

Of course, it’s much harder with compile-time dependencies … If A defines a macro and B uses that to generate code, than probably B should be recompiled after any change to A. I guess that would extend to any other compile-time function calls aside from macros, e.g. module attributes.

@josevalim

It should not take 1 minute to compile 100 files, so I am assuming the slowdown is elsewhere and not in the compiler per-se.

I’m pretty sure the biggest slowdown is from our large GraphQL schema files, which rely heavily on Absinthe macros. That can be seen in the screenshot I shared above with the 2 lines that say Compiling schema.ex (it's taking more than 10s). When I profiled the compilation time, those two files took ~20-25 seconds each to compile.

While I’m sure there’s room for optimizing the compilation time of these individual files, what would be most helpful is if those files didn’t have to be recompiled so often.

@josevalim

Can you please try doing the same IO.inspect change as before and then running:

MIX_DEBUG=1 mix compile --profile=time

and putting the output in a gist? I am particularly interested in the different between these times:

[profile] Finished compilation cycle of 2 modules in 5ms
[profile] Finished group pass check of 2 modules in 0ms
<- Ran mix compile.elixir in 21ms

Here’s the output: Output of `MIX_DEBUG=1 mix compile --profile=time` (filenames anonymized) · GitHub

That part of the output for the 2 longest-running umbrella apps looks like this:

[profile] Finished compilation cycle of 9 modules in 23087ms
[profile] Finished group pass check of 13 modules in 3189ms
<- Ran mix compile.elixir in 26712ms

[profile] Finished compilation cycle of 16 modules in 20364ms
[profile] Finished group pass check of 19 modules in 2366ms
<- Ran mix compile.elixir in 23025ms

The schema.ex files in both those apps are the modules referenced above, the large GraphQL schemas that make heavy usage of Absinthe macros.

Interested to hear whether that is consistent with your expectations, or not.

@josevalim

Btw, if you are interested, please ask your employer if they would be willing to share it with me. If necessary, we could sign a terms of service through Dashbit so we are covered by confidentiality agreements.

Thank you for offering! I’d love to make that happen. My employer previously had a contract with Dashbit, so hopefully it works out. I’m waiting to hear back from others at the company before we can move forward with this.

[josevalim]

Oh, I missed the Absinthe bits. In that case, I am afraid there may not be much to do in Elixir itself. You either need to minimize the compile-time dependencies or someone needs to send PRs to optimize Absinthe (if possible).

[hst337]

@japhib

Of course, it’s much harder with compile-time dependencies … If A defines a macro and B uses that to generate code, than probably B should be recompiled after any change to A. I guess that would extend to any other compile-time function calls aside from macros, e.g. module attributes.

Well, that’s a tricky part. I am working on a compiler where compile-time dependencies are traceable per-function (not per-module as vanilla elixir compiler works right now). But this leads to incompatibility in some very weird edge cases (like someone expects module to be recompiled and inserts like @attr Date.utc_today(), but it is not getting recompiled).

That’s why I brought a proposal of compiler specification. If you’re interested, you can leave your opinion about this situation there

@japhib

While I’m sure there’s room for optimizing the compilation time of these individual files, what would be most helpful is if those files didn’t have to be recompiled so often.

Yeah, that’s another problem caused by dependency granularity in tracing mechanism.

[japhib]

I’ve tried removing the compile-time dependencies in this project, but it appears that run-time dependencies are enough to create this whole chain of re-compilation anyway. (It’s very possible I’m just lacking some understanding of how dependencies work, but in any case, it’s pretty difficult to remove existing dependencies in a large production app.)

@josevalim Would the following optimization be possible in the Elixir compiler?

@japhib

When recompiling a module, if it’s a runtime dependency (C calls B.foo(x, y)), keep track of the exported functions & their arities. If the exported functions/arities don’t change when recompiling, then there’s no need to recompile modules with runtime-only dependencies on that module.

It seems like this would be fairly easy to implement, since Elixir already keeps track of modules that have changed, as well as a map of their dependencies.

I guess maybe it wouldn’t help in cases where the compiler will eagerly compile everything that might be affected, as for a single app where all changed files are re-compiled in parallel. But for a large umbrella app (such as the one I’m working on), where each umbrella app is re-compiled in sequence, this optimization could prevent a lot of re-compilation across app boundaries.

Call me an idealist, but it doesn’t seem like adding an IO.inspect() in a module about 4-5 dependency-modules’ distance away from a GraphQL schema should cause the GraphQL schema to be recompiled every time.

[hst337]

What exactly is the nature of these dependencies? Is it compile-time dependency (like your module can be called from schema module at compile time) or what?

[josevalim]

@japhib

@josevalim Would the following optimization be possible in the Elixir compiler?

We already track these dependencies as export dependencies. Although I need more information/context than the small example above to understand how you propose them to be used.

[japhib]

I’ve working on putting together a minimal umbrella app that reproduces the issue, but it’s not as straightforward as I thought In the process, I’m learning about compile-time dependencies. Maybe I’ll be able to improve our situation by fixing some of the compile-time dependencies in our project.

[japhib]

In the meantime, I do have a question – when running mix compile, it’s easy enough to see which files are getting compiled, with --verbose or --profile=time, but I can’t figure out how to tell why each file is getting recompiled. Is there any flag that would tell me something like that? For instance, something like this:

Module A being compiled because of changes
Module B being compiled because of runtime dependency on A
Module C being compiled because of compile-time dependency on B
...

Does such a compiler flag exist? If not, is there any chance you could point me to the general area in the Elixir codebase where I could add logs to try to deduce this myself?

(Note: I’m aware of mix xref trace and mix xref graph and have been using those tools to track down dependencies. However, even after changing certain dependencies from compile-time to run-time, certain files are still getting unexpectedly re-compiled on each change. The xref tools don’t show any dependencies between the files that would necessitate re-compilation, yet they’re getting recompiled anyway.)

[axelson]

@japhib

Does such a compiler flag exist? If not, is there any chance you could point me to the general area in the Elixir codebase where I could add logs to try to deduce this myself?

(Note: I’m aware of mix xref trace and mix xref graph and have been using those tools to track down dependencies. However, even after changing certain dependencies from compile-time to run-time, certain files are still getting unexpectedly re-compiled on each change. The xref tools don’t show any dependencies between the files that would necessitate re-compilation, yet they’re getting recompiled anyway.)

mix xref graph is the main tool for tracking down compile-time dependencies.

If you haven’t seen it https://medium.com/multiverse-tech/how-to-speed-up-your-elixir-compile-times-part-2-test-your-understanding-f6ff3de5eb5d is a great article that really walks you through what causes a compilation dependencies, including the surprising/tricky bit about transitivity.

Another tool you could try is DepViz, which I wrote when I encountered the same pain that you’re encountering: DepViz - A visual tool to understand inter-file dependencies - #5 by axelson

DepViz will give you a view similar to what you’re asking for. Imagine you’re trying to figure out why create_connection.ex is recompiled, you hover over that file:

From that you can see that there’s 16 files that will cause create_connection.ex to be recompiled. Of those, 3 are direct dependencies, but 15 of them are transitive dependencies via the compile-time dependency on command.ex
(side-note: something is not quite right in the math there! Probably the other two top-level files are being double-counted because there’s a comptile-time dependency loop)

If you want to dig further into how a specific file is reached via command.ex then first click on create_connection.ex, then click on a file (e.g. write_local_name.ex):

From that we can now see that there is the dependency chain:

• create_connection.ex
• command.ex
• write_local_name.ex

Where create_connection.ex depends on command.ex at compile-time, and command.ex depends on write_local_name.ex (at compile-time as indicated by the red dotted lines, although a run-time dependency between these two files would also create a transitive compile-time dependency for create_connection.ex).

I hope this helps you track down your compile-time dependencies!