Excessive recompilations when nothing substantially changed

Hi,

I’m trying to track down why my testing experience is such a pain because there’s always many files being compiled… I’ve used the inotify trick from Understanding Elixir's recompilation · Milhouse to see which files these are (is there a simpler way?) and tried to understand from mix xref why those files were compiled.

So there’s runtime, compile time and exports dependencies, I learned from mix help xref. So when I do a trivial change like adding a newline in a source file, only compile time dependencies should trigger a recompile.
If I change file A, all files having a (transitive) compile time dependency on A would need to be recompiled.

This lead me to use mix xref graph --label compile --only-direct because my reasoning is that this would show me all non-transitive compile time dependencies, and I should be able to traverse from the changed file to every compiled file via a graph node connection in the output of this command.

Here’s a snippet from the output, the email.ex file has only one dependency shown.

lib/my_app/email.ex
└── lib/my_app/views/mailer_view.ex (compile)

The email file was recompiled when I made a trivial change in a file which, according to mix xref (with the invocation above), was not connected at all to the email file.

I must have some misconception here.

If A depends on B at compiletime, then any transitive runtime dependency of B is a compiletime dependency of A. As As compiletime is Bs runtime.

If you are using Bamboo (guessing based off the email view), it was a pretty bad offender in that regard. Luckily the latest release improves the situation significantly:

Now all of your email-related modules won’t recompile all the time (or maybe it’s vice versa).

1 Like

Thanks, that helps a bit.

Thank you, that makes sense, I had not thought of that.

There’s a pattern when writing macros to write as little code as possible inside the macro itself, but use functions as much as possible. The simplest such scenario is calling include to import functions into the calling module. This idea has lead to code like this:

defmodule Fancy do
  defmacro __using__(_) do
    quote do
      # do some stuff...
      import unquote(__MODULE__)
    end
  end

  def utility_function do
    # calls other module
    M.other_fun()
  end
end

When A uses Fancy, we have a compile time dependency A => Fancy
Due to the function inside the Fancy module, there is also a runtime dependency Fancy -> M

So when M changes, A needs to be recompiled.

In my case, A had a transitive runtime dep to the Phoenix router (for calling helpers), which created a huuuuuge recompilation loop.

The easiest solution here is to break the utility functions in Fancy out into a separate module (and file).

I still have the problem that when M is changed, my router gets recompiled, but the only relevant dependencies seem to be these runtime dependencies:

Router -> Controller -> M -> Router

I know a cycle might be bad, but since there is no compile time dependency involved I don’t get why the Router is recompiled.

Is the project public, or can you reproduce it in a fresh project that you can make public?

I’m afraid it’s not public. I think trying to reproduce it from bottom up is the right approach. Don’t know when I can get to that…

Thanks for the advice, I quickly found the culprit:

I had compile time dependencies from the router to two controllers. Because those controllers had a transitive run time dependency back to the router (redirects using route helpers) and from there to every other controller, changing any controller required the router to be recompiled.

The fix:

# change this:
get "/", MyApp.FooController, :index
# to this:
scope alias: MyApp do
  get "/", FooController, :show
end

simply because now the fully qualified module name of the controller it no longer present in the router.

Two other things I did/tried:

elixirc_options: [verbose: true]

in the project/0 callback of the MixFile, which made the effective dependency chain more obvious.

I also tried setting

config :phoenix, :plug_init_mode, :runtime

not only in dev mode, but in test mode as well, assuming that plug initialization would otherwise happen at compile time and that that would also be the case for the controllers being routed to (since they are plugs as well), but I was wrong here, the default :compile does not make all controllers compile time dependencies of the router.

3 Likes

I got tired of analyzing the output of mix xref so I wrote a thing… If you run into this problem please check out GitHub - schnittchen/why_did_recompile and help me improve it

3 Likes

Really neat, I’ll give this a go.

…And I tried it. Does what it promises. My problem is that the project I am participating in is huge (1100+ Elixir files) and the output isn’t giving me a direct feedback what to address because the transitive dependency chains are like 40+ nodes. :frowning:

I used verbose compilation and just touch-ing files and I despaired. :smiley:

Wish Elixir had compiler apparatus to tell you which exact statements in the source file lead to compile / runtime dependency between files. :expressionless: Some of the files are huge, there are use statements, there are macros, there are Phoenix peculiarities (dependencies between router and views and various utilities) so you can very easily get lost in all this.

Not your fault, of course, the tool is useful! Thank you for it.

This sounds oddly familiar to an issue I experienced on an Erlang project. The project was passing macros to erlc at compiletime with -d, and due to a dependency chain it caused a whole-app recompile for each change. Could something similar be happening with your Elixir project? This escaped xref and other tools completely.

I noticed that a simple alias M can make the containing file compile-time depend on the file defining M.

This might be legitimate, but if not, making the compiler not mark the aliased modules’s file a dependency would drastically reduce overall cases.

1 Like

Have you given DepViz a try?

I spoke about it at Code BEAM BR last year (in english): https://www.codebeambr.com/video/12

That is not quite accurate, an alias will only cause a run-time dependency. However, compile-time dependencies are transitive so if A->compile-time->B and B->run-time->C, then when C changes, A will need to be recompiled.

3 Likes

Hah, love the name!

Just did. It’s super laggy though. :frowning: I think it could be very useful for smaller projects.

I am still trying to learn these and what does an “export” or “runtime” dependency even mean. :smiley:

Did you have any success in finding a strategy for diagnosing your issue?

I’m in a similar situation, working on a large project with a ton of files. In some cases changing the css class in a template results in hundreds of files being recompiled. Turning on verbose compilation is somewhat helpful but as you say trying to track things down gets overwhelming quickly!

Nope, unfortunately gave up. The company I worked for back then was extremely output-driven so doing R&D was frowned upon unless you achieve results in a day or two which is unrealistic for such a hard and exploratory task. We parted ways soon after.

Since then I haven’t worked with bigger code bases in Elixir – only in Rust. Those I work with in Elixir are small to medium.

I’d love to work on this again one day – there’s a potential to improve things by piping several CLI tools at the tail of mix xref ... – but between my constant fight to undo a potential serious health condition and working part-time, it seems it ain’t happening for now.