Collect modules that `use` some other module at compile time

Questions / Help
,
1

Hello,

I need some help with the following problem (Bascially I want to store at compile time all modules using some other module, along with their opts).

defmodule Root do
  @modules %{} # Mapping from __MODULE__ to opts

  def __using__(opts) do
    quote do
        @opts unquote(opts)

        def hello(), do: "Hello from #{__MODULE__}"

        # How do I store __MODULE__ and its @opts
        # inside Root.@modules?
        # ...
    end
  end

  def modules_using_me: @modules
end

defmodule A do
  use Root, id: A1
end

defmodule B do
  use Root, id: B1
end

Then to get the modules __using__ Root I would call

iex> Root.modules_using_me()
%{A: [id: A1], B: [id: B1]}

OR

iex> SomeOtherModule.modules_using_root()
%{A: [id: A1], B: [id: B1]}

Could anyone give me some ideas? I already tried to accumulate into a Root attribute all other modules, but obviously it fails since Root module is already compiled.

Thank you!

2 Likes
2

As you observe, Root cannot know what modules use it at compile time, because it must first be compiled itself. Indeed, at compile time only module A could know if it used Root.

You can leverage module attributes to ask A which modules it has used—example code for that here.

Alternatively, since A knows what it’s used, you could inject that information into other modules that use A if you take control of its __using__ macro. This allows a module that uses both A and B to know what modules they respectively have used. This is pretty abstract and hard to explain—I have example code demoing the concept here.

3 Likes
3

Thank you for your reply @christhekeele. Unfortunately it doesn’t address my challenge. I need a way to record all the modules that are __using__ the Root module. Also, these modules are not to be used any further like in your second demo.

4

I’m not sure that’s possible. When Root.__using__ is expanded, the only context it has access to is the module using it (A in our example). So any tracking you want to do must occur within A.

There are only two ways I’m aware of to try to pass information A knows about to modules at compile time: using a macro defined in A (like giving it its own __using__ macro and using it elsewhere); or a @before_compile macro callback (which must come from an already compiled module as well).

I understand your challenge pretty well—since I developed both of those examples a couple of years ago grappling with the same problem. Despite a lot of metaprogramming since then, I haven’t been able to find a way to accomplish exactly what you want in pure Elixir, and those techniques sufficed.

Ultimately I’m pretty sure the problem is intractable, because of the core problems:

  • Root must be compiled to be used
  • only __using__ modules know when Root is used

So all you can really do when Root is used is to push that information around. You can put it in A, or use other techniques to put it in other modules, but by definition and the need for an acyclic module graph, you cannot ever put that information into Root itself.

5

That isn’t to say you couldn’t use techniques outside of pure compile-time Elixir macros to store this information somewhere. The obvious example that comes to mind is a file somewhere. Alternatively, the closest data store adjacent to pure Elixir that is available when the Elixir compiler is executing is :ets. However, no matter how you handle it you run in to the issue where whenever during compilation you need the compiler to access that information, you can’t be fully sure that all modules that would contribute to it have finished compiling.

6

I don’t need the list of modules at compile time (although that would be a bonus) but rather at runtime. I have a GenServer that needs this information. Let’s call it GS. I tried to inspect all loaded modules at GS's init via :code.all_loaded, call some special function I inserted inside Root's __using__, however it looks like not all modules are loaded at the time GS is initialized, thus I don’t get a complete picture. Note: GS is started in one app, while A and B in another, probably that’s why GS doesn’t see all loaded modules.

7

Your problem sounds similar to protocol consolidation. You might find some inspiration in https://github.com/OvermindDL1/protocol_ex which has a custom mix compiler task that scans the beam files for modules that declare implemtations of a protocol.

2 Likes
8

I can easily add a function that returns all implementations of the given protocol too (I think I have it in a descriptor but it’s not really ‘public’ stuff?), but if that is the only feature you are needing then the normal elixir protocols already has that feature too.

But yeah, the scanner part itself could easily do that, copy it into your project and just have it be a compiler that generates the information necessary after everything else is already compiled. :slight_smile:

2 Likes