ElixirConf US 2018 – Understanding Elixir’s (Re)compilation – Renan Ranelli
Elixir’s code-generation capabilities require a sophisticated compiler with complex dependency tracking. Given such complexity, it is often unclear why sometimes changing a single line in a single file triggers the recompilation of 100 other files. This talk aims to clarify that.
Most of the content presented in this talk was “discovered” while struggling with recompilations of 500+ files in a 2000+ .beam file Phoenix app. We learned things the hard way so that you don’t have to.
In this talk we are going to take a deep dive into what happens when you type “mix compile”, why and when modules need to be recompiled, and how compilation behavior interacts with umbrella apps. You will learn how to “debug” recompilation problems, which tools to use, and how to avoid common pitfalls.
I like this talk and I think it’s good but it still doesn’t give me enough insight into our current compilation problems in a Phoenix project. We’ve ended up with a smaller version of the cited issue in the talk and recompilation currently takes 4-5 seconds which is enough to annoy us.
About the struct dependency bit; the speaker says this was changed. Does anyone know exactly how a struct dependency currently behaves in terms of recompilation? Does the Elixir compiler do analysis of struct-only changes or is it simply recompiling the module always regardless?
If your module A only depends on the struct of a module B, then we will only recompile A if the fields in the struct B change. But we always recompile the whole file the module is at. The beam of a module is always assembled as a unit.
A lot of it still comes from things like plugs and phoenix helpers, which are effectively fixed in the next Phoenix v1.4 version.
What I always wondered. If the compiler knows when to recompile which module, couldn’t that information be surfaced to the user? Like a “list me all modules, which are recompiled if module x changes (and why)”. mix xref is alreally nice, but even with all the knowledge about why and when modules depend on each other and need recompiling it’s still quite a lot of work to mitigate things in this space. I’ve had a few shots in a current project of mine and I while sometimes I found the spots to change I also had quite a few times, where I didn’t even find why modules would depend on each other even with the data xref produced. Maybe I should just try updating it to phoenix 1.4 and see how much it helps.
As I mention xref is already doing much, but I found it still hard to go from two modules being listed as dependant on one another to which part of my code is causing this dependency. Even more so if there’s no direct dependency, but rather up to a handful of intermediate modules in between. I’ve had cases where I sat in front of the xref output and wondered “how the **** are those things related to each other in my code”, where modules were handling totally different domains.
The sink option can be really useful for you to find how two things supposedly related to each other. For example: mix xref graph --sink lib/ecto/query.ex in Ecto will show all the paths that reach lib/ecto/query.ex. Some of those are quite long but it does provide the relationship information.
We could also add functionality to provide all paths between two places in particular. Right now it is not possible to filter with a source and sink at the same time.
That would be really helpful I imagine, because if one is really trying to reduce compile time coupling it just comes down to break apart sections at specific edges. The other thing useful, which was mentioned in the talk would be to have a flag to not only filter dependencies by compile/struct/runtime dependencies, but also have a way to have xref list all modules, which will be recompiled a.k.a. compile time dependencies and runtime dependencies, with are implicitly compile time dependencies. I’m not sure if something’s already possible in that direction.
