Building a map of macro list locations during macro resolution

Questions / Help
1

I was interested in trying to collect a list of the __CALLER__ locations throughout a code base where a macro is used. I’ve been able to use :ets to store the information while the macros are being expanded, but once compilation finishes that erlang VM shuts down and I lose the table. I’ve tried to use Module.put_attribute, but whenever I call it elixir tells me the module in question is already compiled.

My goals are:

  • build datastructure containing marco invocation information
  • have that datastructure available at runtime in some form (define a function that returns it, put it in a module attribute, etc)

Right now I have the full datastructure in an :ets: table at the end of macro expansion and can’t figure out how to save it. The module containing the macros is already defined and I can’t put a new attribute on it. Is there a way to call a macro in a module after all other macros are expanded (i.e. to create a function that returns the unquoted copy of the data)?

2

Having tried to make this work on multiple occasions my advice would be to not do it and rather let users provide a list of modules to whatever logic you intend to feed this into. This will likely be more flexible as well. Though I don’t know your usecase, so take the advice with the appropriate grain of salt.

2 Likes
3

Hi there,
There is a compile callback called @after_compile, you can read more about it in the Module module.

If you are looking to store permanent information I could recommend you CubDB, I am using it and it works really well. GitHub - lucaong/cubdb: Elixir embedded key/value database

4

Thank you for sharing your experience, but manual input wouldn’t be practical here. I’m interested in recording the line numbers and other fine details. I was hoping there was a pat solution.

5

Ah, thank you for two solid leads! I’ll check both out.

Edit: unfortunately, @after_compile is local to the modules’ compilation, so it can’t help me here. The module containing the macros (and thus getting the __CALLER__ information) must be compiled in order to resolve the macros. I could do something after expanding all macros in the child modules, but it would be so baroque that using CubDB or some other serialization solution probably works better.

1 Like
6

Well. if you are already saving it to the ETS but it gets lost, then CubDB solves that problem
I would like to see how you are achieving this. So it you don’t mind sharing a module, I could have a look and give you my 2 cents.

1 Like
7

Sure! Here’s an abbreviated example. I’m exploring making a logger helper with richer metadata and runtime control on output options:

defmodule Common.Log.Service do
  use GenServer
  [...]

  def handle_continue(:load_logs, state) do
    locs = Common.Log.locations()
    IO.inspect(locs, label: "log locations")
    {:noreply, state}
  end
end

defmodule Common.Log do
  [...]

  defmacro debug(line) do
    loc =
      Common.Location.create(__CALLER__)
      |> Common.Log.register_log_location(:debug)

    quote do
      Common.Log._make_log(unquote(Macro.escape(loc)), unquote(line))
      |> Logger.debug()
    end
  end
  [...]

  def locations() do
    handle = get_db_handle()
    CubDB.get(handle, :locations, %{})
  end

  defp get_db_handle() do
    if :ets.whereis(:log_locations) == :undefined do
      :ets.new(:log_locations, [:set, :protected, :named_table])
    end

    case :ets.lookup(:log_locations, :db) do
      [] ->
        {:ok, db} = CubDB.start_link("priv/cubdb/logs")
        :ets.insert(:log_locations, {:db, db})
        db

      [{_, info}] ->
        info
    end
  end

  def register_log_location(location, atom) do
    handle = get_db_handle()
    old = CubDB.get(handle, :locations, %{})
    vew_val = update_module_map(old, location, atom)
    CubDB.put(handle, :locations, vew_val)
    location
  end
end

This code is messy, but it accomplishes the basic task. Here’s the inspect of the locations at runtime:

log locations: %{
  App.Application => %{
    start: %{
      8 => {%Common.Location{
         arity: 2,
         file: "/[...]/application.ex",
         function_atom: :start,
         line: 8,
         module: App.Application
       }, :debug}
    }
  }
}

I suppose my answer is “use a file” - but if there’s a way without needing to create a scratch file I would love to know.

8

It’s a little more complicated, but one option would be defining your own compiler (see Mix.Task.Compiler — Mix v1.12.3) and reading module attributes from the compiled BEAM files (how mix compile.protocols — Mix v1.12.3 does it).

I’m working on hacking together a library to do that work (no plans to publish it to hex just yet), but that computer is in the shop right now, and I don’t have it on GitHub yet. But I do have a related experiment I did you could look at (this one doesn’t write out a new BEAM file): activation_elixir/compile.activation.ex at master · brettbeatty/activation_elixir · GitHub

1 Like
9

I would add this to your Application module, son the database is created when you start your supervision tree. So you can dispose of ETS.

I suppose my answer is “use a file” - but if there’s a way without needing to create a scratch file I would love to know.

CubDB is actually your file.

10

I would add this to your Application module, son the database is created when you start your supervision tree. So you can dispose of ETS.

Is there a way to interact with CubDB without the handle returned by start_link? It wasn’t clear from the documentation.

CubDB is actually your file.

I know it is - I was saying I would like a solution that doesn’t generate an extra file.

11

Yes. You create a DB module which is a GenServer that wraps CubDB calls.
Later today I will upload the source code of an app where you can see how this is done.

12

Yes. You create a DB module which is a GenServer that wraps CubDB calls.

Unless I’m not following, you’re saying that you do need the handle returned by start_link - but you are suggesting wrapping it in a GenServer? I do not think that will work for my scenario because application supervision trees are not started during macro expansion - I will still need to manage the state within the file and :ets.

For other people it would absolutely make sense to wrap it in a GenServer once you start the application proper. I did not do that in the above example because I’m only using CubDB to bridge across the compile / runtime divide.

13

You are right. I totally overlooked the fact that you are doing all this at compile time

1 Like