Getting a list of "tagged" modules

Easiest to ask this in code…

defmodule Foo do
  use Tagger
  tag(:red)
end

defmodule Bar do
  use Tagger
  tag(:blue)
end

Then at runtime I want to be able to say:

Tagger.all_tagged # %{ Foo => :red, Bar => :blue }

How to define Tagger to do this? I know how to use module attributes and macros to create methods on Foo and Bar to keep track of what they are tagged with, but how to aggregate that info into another module?

Thanks for the help.

2 Likes

I would probably use module attributes to store the tag in the compiled module:

defmodule Tagger do

  defmacro __using__([]) do
    quote do
      Module.register_attribute(__MODULE__, :tag, persist: true)
    end
  end

end

defmodule Foo do
  use Tagger
  @tag :red
end 

You can then inspect the attributes using module_info/1:

iex(1)> Foo.module_info(:attributes)[:tag]
[:red]

Given a list of modules you can then simply map, filter, aggregate, etc. using Enum.

As for the list of modules, you can use {:ok, modules} = :application.get_key(:my_app, :modules) to get all modules for your OTP app. Or, if you want to scan for modules more broadly, you can look at iex’s autocomplete for inspiration:
https://github.com/elixir-lang/elixir/blob/master/lib/iex/lib/iex/autocomplete.ex#L247

7 Likes

Oh wow. A lot more complicated than I thought, but this solves it! Thanks a ton!

1 Like

Just for posterity, the :code.all_loaded() technique didn’t work for me. I’m not sure what marks a module as “loaded”, but when I started an iex console and called :code.all_loaded(), my modules didn’t show up in the list.

Here’s what I did instead:

Enum.each :application.loaded_applications(), fn {app, _, _} ->
  {:ok, modules} = :application.get_key(app, :modules)
  Enum.each modules, fn mod ->
    # Do my thing with mod
  end
end

Now what marks an application as loaded? I have no idea. But :application.loaded_applications() seems to return all applications in my mix project… at least at the point where I’m calling it (which is the mod: [] callback).

1 Like

It works for any code that is indeed loaded. It is considered loaded when some function on the module is called, until that point it is not loaded (unless the erlang compiler option in the boot forces loads everything in the one file). Your modules will appear once you call something on them.

An application is loaded by the boot loader, but again that gets back into manual plugin definitions instead of dynamic detection. ^.^

5 Likes

If you look closely you’ll see that the IEx code only relies on :code.all_loaded() alone when :code.get_mode() returns anything other than :interactive.

When you start your application as part of a release, the :embedded mode is used, and all modules included in the release a loaded right after the VM has started. On the other hand, when you run iex on your Mix project, the code server uses :interactive mode in which modules are located and loaded on-demand, as @OvermindDL1 described.

As for application loading, iex -S mix loads your main application and all its dependencies (actually, it first loads the dependencies and then the main app). Some dependencies may explicitly load/start additional applications as they initialize.

5 Likes

I would probably use module attributes to store the tag in the compiled module:

I was also thinking of using module attributes to store the tags (that’s how I found this thread), but eventually I dropped the idea. The reason was that (according to documentation) stripping debug symbols removes all module attributes:

The list of attributes becomes empty if the module is stripped with the beam_lib(3) module (in STDLIB).

I haven’t tested it, but if it is so, then it may happen that the code appears to work, until somebody decides to strip debug symbols (e.g. while releasing to production) and then the code mysteriously breaks. So using this approach at runtime it potentially dangerous.

1 Like

Somewhere over the last two years, I noticed that Elixir records behaviours in module attributes, which is actually the use case I was going for. So here’s the code I use now:

{:ok, modules} = :application.get_key(:my_app, :modules)
Enum.map(fn module -> module.__info__(:attributes)[:behaviour] end)

Which apparently relies on the debug info not being stripped to work… :confused: Thanks for the tip!

Any ideas to work around the problem?

Forgot to mention… that above code snippet works when using Elixir releases.

Depending on how fast you want to make this you could also try to do all this work at compile time.

But that carries it’s own compilations, you obviously want all modules using Tagger which means you somehow need to wait until everything is compiled.

I’ve found this thread on the matter but it seems this is no simple endeavour.

1 Like

I think the best option is to use @before_compile hook for tagging. Check the example below:

defmodule Tagger do
  defmacro __before_compile__(_env) do
    quote do
      def __tagged, do: :ok
    end
  end

  def list_tagged_modules do
    {:ok, modules} = :application.get_key(Application.get_application(__MODULE__), :modules)
    modules |> Enum.filter(fn m -> m.__info__(:functions) |> Enum.any?(&match?({:__tagged, 0}, &1)) end)
  end
end

defmodule Mod1 do
  @before_compile Tagger
end

defmodule Mod2 do
end

And this is how you use it:

iex(1)> Tagger.list_tagged_modules
[Mod1]

So effectively “@before_compile Tagger” does the tagging.

1 Like

Note that the hook injects a method __tagged(), and then list_tagged_modules() checks for the presence of the method in module. So it should always work, unless there is a clash with an existing method. That’s why the name of the method must be unique.

Elxir Protocols have an interesting reflection feature. It looks like SomeProtocol.__protocol__(:impls) and can return a {:consolidated, modules}. It shows all modules, that implement the protocol. You can use this to tag modules, like:

defprotocol TagExample.Tagged do
  @spec tags(t()) :: [atom()]
  def tags(data)
end

defmodule TagExample.Tag do
  alias TagExample.Tagged

  defmacro __using__(_opts) do
    quote do
      Module.register_attribute(
        __MODULE__,
        :tag,
        persist: true,
        accumulate: true
      )

      defimpl TagExample.Tagged do
        def tags(_) do
          :attributes
          |> @for.__info__()
          |> Keyword.get(:tag, [])
        end
      end
    end
  end

  defimpl Tagged, for: Atom do
    def tags(module) do
      :attributes
      |> module.__info__()
      |> Keyword.get(:tag, [])
    end
  end

  @spec tagged_modules() :: [module()]
  def tagged_modules do
    modules =
      case Tagged.__protocol__(:impls) do
        {:consolidated, modules} ->
          modules

        _ ->
          Protocol.extract_impls(
            Tagged,
            :code.lib_dir()
          )
      end

    for module <- modules,
        module.__info__(:attributes)
        |> Keyword.has_key?(:tag),
        do: module
  end

  @spec modules_with_tag(tag :: atom()) :: [module()]
  def modules_with_tag(tag) do
    for module <- tagged_modules(),
        module
        |> Tagged.tags()
        |> Enum.member?(tag),
        do: module
  end
end

defmodule TagExample.TaggedModule1 do
  use TagExample.Tag

  @tag :tag1
  @tag :tag2
end

defmodule TagExample.TaggedModule2 do
  use TagExample.Tag

  @tag :tag2
  @tag :tag3
end

# iex> TagExample.Tag.tagged_modules()
# [TagExample.TaggedModule2, TagExample.TaggedModule1]

# iex> TagExample.Tag.modules_with_tag(:tag2)
# [TagExample.TaggedModule2]

# iex> TagExample.Tag.modules_with_tag(:nonexisting_tag)
# []

This example uses module attributes, but you can store tags somewhere else (possibly in methods).

I have also created a blog post with edge cases for this implementation.