Excessive recompilations when nothing substantially changed

I recently dealt with a larger project that had these problems.

This info is scattered around these forums and I know someone wrote a blog post recently but I can’t find it.

Basically the main culprit is pretty much always cyclical dependencies between modules and, depending on how intertwined they’ve gotten, it can be really hard to fix.

Basically look for things like this:

defmodule A do
  def a, do: "a"

  def b() do
    B.b()
  end
end

defmodule B do
  def b, do: "b"

  def a() do
    A.a()
  end
end

Even if they don’t explicitly cause a compile-time deps (my example won’t), you want to avoid code like that in general. It gets tricky with Elixir since some things were are normally just runtime deps can become compile time deps if the thing they depend on has a compile time dep. For example if I were to introduce the following:

defmodule C do
  @b B

  def b, do: @b.b()
end

…now both B and A have a compile-time dep on C (with A also depending on B). I won’t get into why this is to keep this post shorter, but avoiding cyclical dependencies will avoid this scenario altogether. (I’m sure someone is going to correct me on me calling them cyclical but I’m not sure what else to call them).

The solution is to always(*) design your module hierarchy as a DAG. Ensure that children never talk to their parents and that when siblings talk it only goes in one direction (and never loops).

(*) of course there are exceptions to every rule

Anyway, sorry if I’m telling you things you already know, but that is where I would look first. People often jump to eliminating the easy targets (like removing references to modules within other module bodies). Those help, but honestly, having a few compile time deps isn’t so bad so long as they are in files that aren’t often changed.

Unfortunately I can’t remember the exact xref commands I would use, but this was a common one:

$ mix xref graph --sink SOME_FILE_THAT_GIVES_YOU_HECK.ex --label compile

…where SOME_FILE_THAT_GIVES_YOU_HECK.ex is a file that causes tons of recompilations. This will show you all the files you have with compile time deps on that file. If you remove the --label compile then you should get an insanely large tree. It’s useless to look at the whole thing but what you want to do is look for repeating files, like so:

a.ex
|-b.ex
|-|-c.ex
|-|-|-d.ex
|-|-|-|-e.ex
|-|-|-|-|-a.ex

This situation can be very hard to detangle, especially if a.ex appears, nested, again.

1 Like

Also, though, just avoid calling modules from other module bodies. For example, there was a lot of this in the codebase I worked in:

defmodule Foo do
  @foos = %{
    a: A,
    b: B,
    c: C
  }
end

Don’t do that!

Sorry to hear about the health issues, hope they get resolved!

Nice! Thanks for the suggestions, definitely gives me some things to look into.

1 Like

Not quite, you have that reversed. Now C has a compile-dependency on both A and B. The compile-time dep on B is direct, but the compile-time dep on A is transitive (because A depends on B at compile-time). This means that if the a.ex file or b.ex file is changed then c.ex needs to be recompiled (it generally doesn’t make too much difference but compilation dependencies are tracked on a per-file basis even though the dependencies are introduced on a per-module basis).

I’m pretty sure this is the blog post you are referring to:
https://medium.com/multiverse-tech/how-to-speed-up-your-elixir-compile-times-part-2-test-your-understanding-f6ff3de5eb5d

I highly highly recommend it to anyone struggling with recompilation in Elixir. It’s well worth the time to read through and test out the exercises.

Also a shameless plug for anyone that wants to narrow down where they might want to focus their recompilation efforts I wrote a tool called DepViz:

2 Likes

One thing that got used a bunch at my old company was repurposing behaviour modules for other things. For example, a lot of them would expose a function to enumerate all the modules that implemented the behaviour (which is not a good idea when you think about it). Anyway, that was one more thought I had that might possibly be in your codebase.

Oops, yes indeed! And yes, that is the post, thanks! I’ve saved it this time 'round.

1 Like

Was revisiting this a bit today and realized I forgot to mention that in addition to the sugggestions above we found adding GitHub - sasa1977/boundary: Manage and restrain cross-module dependencies in Elixir projects to be super helpful. It helped surface some poorly defined boundaries we had going on.

3 Likes