Compiler cycles in Phoenix

Our Elixir/Phoenix project is growing, and our re-compilation has been suffering. So I started using the amazing mix xref tool to identify in particular cycles in code.

I am now down to very few cycles, and most of them seem to be because of how we use Phoenix. However I think we use it as suggested in the docs. Or perhaps we are doing something very stupid. If so apologies, and thanks for any pointers.

But basically we have about 14 cycles of the following kind. Some longer, but I think it illustrates the point.

Cycle of length 3:

    lib/my_app/controllers/my_controller.ex
    lib/my_app/router.ex
    lib/my_app/controllers/my_controller.ex

Our controller depends on router only through alias MyApp.Router.Helpers, as: Routes. And obviously the router need to depend on controllers, to actually forward requests to the correct functions.

I am not sure how big of an impact these cycles are, but now that I am trying to hunt them down, it would be nice to get rid of them all.

So I guess my first question is, are we doing something apparently wrong? And if not, I guess this is by design? A controller needs to know routes, and these routes are defined in router. And router needs to delegate to controllers… So to me, it kind of seems like it has to be that way?

Thanks,
Alf

1 Like

I think it depends on the type of dependencies between the modules.

Reading mix help xref:

Dependencies types

Elixir tracks three types of dependencies between
modules: compile, exports, and runtime. If a module has a
compile time dependency on another module, the caller
module has to be recompiled whenever the callee changes.

Compile-time dependencies are typically added when using
macros or when invoking functions in the module body
(outside of functions).

Exports dependencies are compile time dependencies on the
module API, namely structs and its public definitions.
For example, if you import a module but only use its
functions, it is an export dependency. If you use a
struct, it is an export dependency too. Export
dependencies are only recompiled if the module API
changes.
Note, however, that compile time dependencies
have higher precedence than exports. Therefore if you
import a module and use its macros, it is a compile time
dependency.

Runtime dependencies are added whenever you invoke
another module inside a function. Modules with runtime
dependencies do not have to be compiled when the callee
changes, unless there is a transitive compile or export
time dependency between them.

I had a look at the largest of my projects (admittedly not too large but has quite a few nested live views and components) and graphing the cycles for each label gave me:

$ mix xref graph --format cycles
27 cycles found. Showing them in decreasing size:
...
$ mix xref graph --format cycles --label runtime
26 cycles found. Showing them in decreasing size:
...
$ mix xref graph --format cycles --label export
1 cycles found. Showing them in decreasing size:
...
$ mix xref graph --format cycles --label compile
No cycles found

So from the dependency type descriptions, if most of my cycles are runtime dependencies I wouldn’t be too worried about them.

3 Likes

Thanks for your help and analysis. And apologies for the late reply, got my second corona shot on Friday, and have been “down” for the weekend.

I’ve managed to remove all our app cycles except the ones caused by phoenix. And re-compilation of a single module has gone down from typically 30-40 seconds to 5-8 second. I suspect if I got rid of these remaining cycles 14 cycles, I would shave off a few more seconds.

But I guess this is not possible then?

We have two app, which arguably should be one. But changing something in “core”, always triggers a recompilation of two modules in “web”. And I suspect these cycles are involved.

The mix xref graph --sink module.ex --only-nodes --include-siblings also do not seem to track across apps in an umbrella project. Which makes it a bit hard to see which modules needs to be recompiled.

When doing “recompile” in iex is there a way to see what is being compiled?

Best regards,
Alf

Hope you’re feeling better!

Short answer is I don’t really know, but here is my train of thought:

IEx.helpers.r/1 returns a map of module names that were in turn returned by Code.compile_file/1. The Code module describes compilation tracers that you can attach to the compiler, where you can handle specific compilation events.

That’s probably your best bet to get a good picture of what’s going on, but that’s as far as my knowledge goes.

Please understand I’m still very much learning and any attempts at answering a question is with the intent of furthering my knowledge as well as hopefully helping someone else gain some insight.

1 Like

You can use file watchers for the fs to watch the beam files.

Runtime dependencies do not affect compilation times. So if your remaining cycles are indeed only runtime dependencies then you won’t get faster compliation by removing them.

3 Likes

Alternatively you could do it outside of iex by invoking mix compile --verbose in your shell (after changing a file first of course). You’ll see exactly what’s getting compiled.

3 Likes

That sounds like a nice flag to add to recompile as well.

2 Likes

We have this same problem at my day job. After reading this tweets from José we created an experimental branch to update Phoenix to the 1.6 development version. This update reduced a lot our compilations, so I would suggest to try the same.
We haven’t updated our application to use Phoenix 1.6 yet, but we plan to do it as soon as it is released.

On the other hand, the new additions to mix xref are great, we have created another experimental branch to use latest Elixir 1.13 development version and set up our CI to throw an error if there is any transitive compile time dependencies. This has uncovered some nasty recompilations that we are currently fixing.
Previously we had some artisanal check that compiled the project, touched a file, compiled again and checked the number of recompiled files. This avoid increasing recompilations, but didn’t provide as much information as the new mix xref options.

The future looks certainly bright and In my experience the new version of Phoenix will improve compilations a lot. The new version of Elixir will make it much easier to find and solve compilation problems.

4 Likes

I believe you meant Phoenix 1.6?

Super useful, thank you! I would be one of the first adopters when it comes out officially. :slight_smile:

Mind posting a permalink (to hexdocs with exact Elixir version) to the new options for forum historical reasons?

2 Likes

Thanks for all the tips everyone!

The mix compile --verbose was especially useful to find what is recompiled between apps in a umbrella project.

We had an idiotic module attribute which depended on a central part of core, which was the always recompiled. But more surprising was that we had a controller with a plug, .e.g:

plug(:put_layout, {Web.EmailView, "layout.html"})

This also made it recompile whenever a dependent module was changed.

Finally what Is left, is that our router is always recompiled. I think I will wait with that until we get til Phoenix 1.6.

The remaining problems are not cycles, but cycles seem to make re-compilation extremely slow. So avoiding them in the first place would’ve better than to get into the mess we’ve been in. Pretty sure our code would have been better designed if we only had tooling help from beginning.

So I thought I’d propose a compiler flag like the --warnings-as-errors but for cycles, .e.g --cycles-as-errors. The compiler knows… I’ve been a clojure programmer for some time, and there namespace cycles are just not possible. I think and opt-in addition to the Elixir compiler would be helpful to many.

1 Like

From Jose’s tweets linked above: come Elixir 1.13 we could use mix xref graph --label compile-connected --fail-above 0 as a CI step, or as a part of a local linting script (I strive hard to have CI replicated on my machine, always). I am very pumped about it. :smiley:

2 Likes

Yes, I meant Phoenix 1.6. I messed up Elixir and Phoenix versions :joy:
I’ve updated the post to remove this error.

Unfortunately hexdocs does not have the 1.13 development version of Elixir (the last version that I see is 1.12.2), so I can’t link to it at the moment.

You can see the docs for the master branch by putting “master” as the version in the hexdocs url: Elixir v1.13.0-dev — Documentation

Once 1.13 is released, it will be available at https://hexdocs.pm/elixir/1.13

That’s not a stable url though. It’ll always point at master even after 1.13 is released.

You’re correct, and I didn’t notice they asked for a permalink :sweat_smile:

Ah, I see. We’ll have a permalink after Elixir 1.13 is released then.