Accessing a module in the main app from a dependency

I don’t think the code being compiled in my_lib can see the code in my_app at all - it would massively complicate dependency tracking if files in the application could cause libraries to recompile.

Have you considered an approach like Ecto.Repo where callers say use MyLib, modules: [...etc...] in a module in my_app? The for loop and friends stay in my_lib but the use of them happens when my_app is compiled.

That’s a brilliant idea! Thank you so much for this. This also solves a future problem I’d have had when a user would like to create two “instances” of my_lib. This way they can just use it multiple times.

One small problem, I’m sure I’m doing something wrong, but I’m running into the error, that conn from Plug cannot be found in my macro:

defmacro __using__(opts) do
  modules = opts[:modules]

  quote do
    use Plug.Router

    plug :match
    plug :dispatch

    get "/" do
      conn
      |> send_resp(200, unquote(modules))
    end
  end
end

== Compilation error in file lib/my_app/module_using_my_lib.ex ==
** (CompileError) lib/my_app/module_using_my_lib.ex:2: undefined function conn/0
    (elixir 1.10.3) src/elixir_locals.erl:114: anonymous fn/3 in :elixir_locals.ensure_no_undefined_local/3
    (stdlib 3.13) erl_eval.erl:680: :erl_eval.do_apply/6

I’ve tried wrapping conn in Macro.var, to no success. But to my understanding that shouldn’t even be necessary in this case.

Not sure what the forum policy on bumping is - if it is allowed consider this topic bumped. If not I’ll go ahead and open a new topic for this second question.

Changing conn to var!(conn) will compile cleanly I believe.

This seems to have done the trick, although I am not sure why this is required in this case. Thank you very much!

Now I am running into a different macro issue entirely:

My __using__ macro takes an opts argument which is a proplist with several options.

Since I want to post-process the options at compile time before inserting them into code I’ve gone ahead and implemented it in the following way (the implementation of normalize_module should not be relevant):

defmacro __using__(opts) do  
  quote do
    modules = unquote(opts[:modules])
    modules = modules |> Enum.map(&MyLib.normalize_module/1)
    
    def get_modules(), do: unquote(modules)
  end
end

This creates an issue in the unquote(modules) line, since the contents of modules is no longer coming from the macro, but its own body:

== Compilation error in file lib/myapp/module_using_my_lib.ex ==
** (CompileError) lib/myapp/module_using_my_lib.ex:2: undefined function modules/0
    (elixir 1.10.3) src/elixir_locals.erl:114: anonymous fn/3 in :elixir_locals.ensure_no_undefined_local/3
    (stdlib 3.13) erl_eval.erl:680: :erl_eval.do_apply/6

• How would I go about inserting the current content of the “modules” variable in this line? The way I did it was the way one would approach it outside of a macro.
• Is this the proper way to implement such a behavior? I also tried implementing it outside the quote part, which came with its own set of issues (including having to use Code.eval_quoted which I’d like to avoid).

Huge thanks to everyone who has been helping me out so far!

Remember that everything in the quote block is actually runtime execution. Outside the quote is compile time execution. Therefore you are probably after:

defmacro __using__(opts) do  
  modules = 
    opts
    |> Keyword.get(:modules)
    |> Enum.map(&MyLib.normalize_module/1)

  quote do
    def get_modules(), do: unquote(modules)
  end
end

Do note however that macros both receive and return AST. So opts[:modules] may not be what you expect since it will be AST.

You may find that you need the following signature for MyLib.normalize_module/1:

def normalize_module({:__aliases__, _meta, [module_name]}) do
  module_name
end

or alternatively:

def normalize_module(module_ast) do
  Macro.expand(module_ast, __ENV__)
end

I see, thank you. That’s quite unfortunate, since this feels like it makes this far more difficult than it has to be.

For example my input AST can look like the following:

[
  {:%{}, [line: 10],
   [
     module: {:__aliases__, [counter: {MyApp.ModuleUsingMyLib, 1}, line: 10],
      [:MyApp, :Plug1]},
     some_opt: true,
     another_opt: {:%{}, [line: 10], [key: 1]}
   ]},
  {:__aliases__, [counter: {MyApp.ModuleUsingMyLib, 1}, line: 13],
   [:MyApp, :Plug2]}
]
# More variants are possible

Since the AST can be and will be deeply nested I’d be unable to use the “standard” elixir functions for interacting with data structures.
Also it feels wrong to make a function (the normalize_module function) dependent on receiving AST when it could potentially called at runtime with non-AST data.

Calling Macro.expand seems to do nothing with my AST.

Is there a better way to do this?

I’d ideally not receive this option as AST but as a “normal” data structure, which contents I can after modifying turn into a literal and insert into the code.

Since macros are primarily about manipulating code, not executing it, I’m not sure how your expectation could be met.

Nevertheless, you can evaluate AST at compile time with Code.eval_quoted/3 although as the documentation says:

Warning: Calling this function inside a macro is considered bad practice as
it will attempt to evaluate runtime values at compile time. Macro arguments are
typically transformed by unquoting them into the returned quoted expressions
(instead of evaluated).

Using your example:

defmacro __using__(opts) do  
  options = Code.eval_quoted(opts)
  modules = Keyword.get(options, :modules)

  quote do
    def get_modules(), do: unquote(modules)
  end
end

The benefit here isn’t really enough to warrant using Code.eval_quoted/3 so perhaps reverting closer to your original idea may be clearer since it too will get resolved at compile time (but in a later compiler expansion):

defmacro __using__(opts) do
  quote do
    modules = unquote(opts[:modules])
    @modules Enum.map(modules, &R.normalize_module/1)

    def get_modules(), do: @modules
  end
end

I see, for some reason I always looked at them as a way of executing code at compile time, but this makes a lot more sense.

True, I never even considered the fact that a user might want to e.g. use a runtime configuration value. Then I guess I’ll budge and do the data transformation fully at runtime.

defmacro __using__(opts) do
   modules = opts[:modules]

   quote do
    def get_modules() do 
      unquote(modules)
      |> Enum.map(modules, &R.normalize_module/1)
    end
  end
end

I initially wanted to avoid this to save performance at runtime, but this doesn’t seem to be a good way to do it. Maybe this is a case of premature optimization being evil again

Thanks for your time and patience. I wish I could mark the answers of multiple people as solution, but I’ll just go ahead and mark your last one.

Have a great weekend!

In my somewhat lengthy reply you might have missed this version which is compile-time calculation of your modules: