Why are compile-time dependencies between modules transitive?

A chain like A ← B ← C can happen very easily:

• A defines a macro
• B uses the macro to generate a function foo/2
• C calls B.foo(x, y)

Changing the definition in A to instead produce a foo/3 and recompiling should fail, but that will only happen if C is recompiled.

There’s a lot more information in the docs for mix xref, which you’ll need to get familiar with to to figure out what’s going on in your application.

Also make sure you’re on the latest Elixir version if at all possible - the core team has done a lot of work behind-the-scenes to try to reduce these kind of headaches in the last couple releases.

There are a couple of good posts explaining some of the issues and how to analyse them.

e.g. Understanding and fixing recompilation in Elixir projects, Implicit compile-time dependencies, https://medium.com/multiverse-tech/how-to-speed-up-your-elixir-compile-times-part-1-understanding-elixir-compilation-64d44a32ec6e

Besides the excellent replies above, there are likely places we can optimize Mix or the compiler. It should not take 1 minute to compile 100 files, so I am assuming the slowdown is elsewhere and not in the compiler per-se. Can you please try doing the same IO.inspect change as before and then running:

MIX_DEBUG=1 mix compile --profile=time

and putting the output in a gist? I am particularly interested in the different between these times:

[profile] Finished compilation cycle of 2 modules in 5ms
[profile] Finished group pass check of 2 modules in 0ms
<- Ran mix compile.elixir in 21ms

My theory is that there is a large overhead on mix compile.elixir compared to the actual compiler. Looking at the code, I can think of some ideas for improvements, so I will try them out now.

Btw, if you are interested, please ask your employer if they would be willing to share it with me. If necessary, we could sign a terms of service through Dashbit so we are covered by confidentiality agreements.

How about this scenario? Given these modules:

$ cat lib/a.ex 
defmodule A do
  def foo(), do: "hey"
end

$ cat lib/b.ex
defmodule B do
  def bar() do
    A.foo()
  end

  defmacro __using__(_opts) do
    []
  end
end

$ cat lib/c.ex
defmodule C do
  use B
end

$ cat lib/d.ex
defmodule D do
  use B
end

I’m wondering if the recompilation cascade could be “stopped” if some module in a dependency chain didn’t change in a meaningful way (its bytecode stayed the same). Currently if I modify A then both dependencies of B get recompiled, even though B didn’t really change.

$ # modify lib/a.ex

$ mix compile --verbose
Compiling 3 files (.ex)
Compiled lib/a.ex
Compiled lib/d.ex
Compiled lib/c.ex

B doesn’t need to change for the change in A to have effects on C and D. Call A.foo within B.__using__() and you have C and D actually depending on A.

Right. So in this case the compiler would have to be much smarter to see that this runtime dependency shouldn’t be transitive to compile dependencies because it’s not used in an exposed macro, tricky…

But then if I add one more level of spaghetti macros:

$ cat lib/aa.ex
defmodule AA do
  defmacro __using__(_) do
    []
  end

  def baz(), do: "hey"
end

$ cat lib/a.ex 
defmodule A do
  use AA
  def foo(), do: "hey"
end

and I modify AA.baz to return hey2, the bytecode of A won’t change, but C and D will get recompiled:

$ mix compile --verbose
Compiling 4 files (.ex)
Compiled lib/c.ex
Compiled lib/d.ex
Compiled lib/aa.ex
Compiled lib/a.ex

Thank you for the information! I will be sure to read the resources provided.

That makes sense. It seems like this particular case would be solved in a fairly straightforward way – when recompiling a module, if it’s a runtime dependency (C calls B.foo(x, y)), keep track of the exported functions & their arities. If the exported functions/arities don’t change when recompiling, then there’s no need to recompile modules with runtime-only dependencies on that module.

Of course, it’s much harder with compile-time dependencies … If A defines a macro and B uses that to generate code, than probably B should be recompiled after any change to A. I guess that would extend to any other compile-time function calls aside from macros, e.g. module attributes.

I’m pretty sure the biggest slowdown is from our large GraphQL schema files, which rely heavily on Absinthe macros. That can be seen in the screenshot I shared above with the 2 lines that say Compiling schema.ex (it's taking more than 10s). When I profiled the compilation time, those two files took ~20-25 seconds each to compile.

While I’m sure there’s room for optimizing the compilation time of these individual files, what would be most helpful is if those files didn’t have to be recompiled so often.

Here’s the output: Output of `MIX_DEBUG=1 mix compile --profile=time` (filenames anonymized) · GitHub

That part of the output for the 2 longest-running umbrella apps looks like this:

[profile] Finished compilation cycle of 9 modules in 23087ms
[profile] Finished group pass check of 13 modules in 3189ms
<- Ran mix compile.elixir in 26712ms

[profile] Finished compilation cycle of 16 modules in 20364ms
[profile] Finished group pass check of 19 modules in 2366ms
<- Ran mix compile.elixir in 23025ms

The schema.ex files in both those apps are the modules referenced above, the large GraphQL schemas that make heavy usage of Absinthe macros.

Interested to hear whether that is consistent with your expectations, or not.

Thank you for offering! I’d love to make that happen. My employer previously had a contract with Dashbit, so hopefully it works out. I’m waiting to hear back from others at the company before we can move forward with this.

Oh, I missed the Absinthe bits. In that case, I am afraid there may not be much to do in Elixir itself. You either need to minimize the compile-time dependencies or someone needs to send PRs to optimize Absinthe (if possible).

Well, that’s a tricky part. I am working on a compiler where compile-time dependencies are traceable per-function (not per-module as vanilla elixir compiler works right now). But this leads to incompatibility in some very weird edge cases (like someone expects module to be recompiled and inserts like @attr Date.utc_today(), but it is not getting recompiled).

That’s why I brought a proposal of compiler specification. If you’re interested, you can leave your opinion about this situation there

Yeah, that’s another problem caused by dependency granularity in tracing mechanism.

I’ve tried removing the compile-time dependencies in this project, but it appears that run-time dependencies are enough to create this whole chain of re-compilation anyway. (It’s very possible I’m just lacking some understanding of how dependencies work, but in any case, it’s pretty difficult to remove existing dependencies in a large production app.)

@josevalim Would the following optimization be possible in the Elixir compiler?

It seems like this would be fairly easy to implement, since Elixir already keeps track of modules that have changed, as well as a map of their dependencies.

I guess maybe it wouldn’t help in cases where the compiler will eagerly compile everything that might be affected, as for a single app where all changed files are re-compiled in parallel. But for a large umbrella app (such as the one I’m working on), where each umbrella app is re-compiled in sequence, this optimization could prevent a lot of re-compilation across app boundaries.

Call me an idealist, but it doesn’t seem like adding an IO.inspect() in a module about 4-5 dependency-modules’ distance away from a GraphQL schema should cause the GraphQL schema to be recompiled every time.