Accessing a module in the main app from a dependency

Questions / Help
, , , , ,
1

Hello there,

I have the following scenario:

I have a library my_lib that is used in my_app. my_lib has a configuration file option that accepts a list of modules. Based on the list of modules provided, a Plug is generated.

The config in my_app is defined like this:

config :my_lib,
  modules: [MyApp.Plugs.Plug1, MyApp.Plugs.Plug2]

The code in my_lib looks like this:

modules = Application.get_env(:my_lib, :modules)
for module <- modules do
  quote do
    forward "/module/#{unquote module}", to: unquote(module)
  end
end

This compiles, but a request to /module/Elixir.MyApp.Plugs.Plug1 just yields a 404 error. Interestingly, I also get a warning in VS Code in the for line, that modules cannot be iterated since it is nil, which to my understanding cannot be the case since I defined a default value of [] for it the mix.exs file of my_lib (I can also certainly confirm that it is not nil, since logging it prints the expected value).

def application do
  [
    mod: {MyLib.Application, []},
    env: [modules: []]
  ]
end

At this point I assumed that I either made an error in my meta-programming or since my_lib is compiled before my_app, that its modules are simply not yet available.
According to the documentation Code.ensure_compiled/1 would be a good candidate to wait for the compilation of these modules. So I added a call to this function and printed out its results, which were quite baffling to me:

modules = Application.get_env(:my_lib, :modules)
for module <- modules do
  compiled = Code.ensure_compiled(module)
  IO.puts("ensure_compiled #{inspect module} -> #{inspect compiled}")
  # = ensure_compiled MyApp.Plugs.Plug1 -> {:error, :nofile}
  # = ensure_compiled MyApp.Plugs.Plug2 -> {:error, :nofile}
  quote do
    forward "/module/#{unquote module}", to: unquote(module)
  end
end

Apparently these modules do not exist at all. However, calling this function directly in IEx, the call resolves successfully:

iex(1)> Code.ensure_loaded MyApp.Plugs.Plug1
{:module, MyApp.Plugs.Plug1}

If I had to take another guess the module resolution fails since my_lib is looking for a source file in its own lib folder and does not take other applications (my_app) into account.

  • Is there any way to use modules of the “parent” app during compilation? If so, how would I go about that?
  • What did I do wrong that Plug did not throw an error or return a 500, but just silently ignore my module (well, or just works for that matter :smiley: )?
  • Did I set the default config value incorrectly? It seems to work when executing, but since VS Code (using the elixir-ls extension) complains about it, I guess there must be something not quite right with it.

Thank you for your time! I realize that this is quite the wall of text :smiley:

2

I don’t think the code being compiled in my_lib can see the code in my_app at all - it would massively complicate dependency tracking if files in the application could cause libraries to recompile.

Have you considered an approach like Ecto.Repo where callers say use MyLib, modules: [...etc...] in a module in my_app? The for loop and friends stay in my_lib but the use of them happens when my_app is compiled.

1 Like
3

That’s a brilliant idea! Thank you so much for this. This also solves a future problem I’d have had when a user would like to create two “instances” of my_lib. This way they can just use it multiple times.

One small problem, I’m sure I’m doing something wrong, but I’m running into the error, that conn from Plug cannot be found in my macro:

defmacro __using__(opts) do
  modules = opts[:modules]

  quote do
    use Plug.Router

    plug :match
    plug :dispatch

    get "/" do
      conn
      |> send_resp(200, unquote(modules))
    end
  end
end

== Compilation error in file lib/my_app/module_using_my_lib.ex ==
** (CompileError) lib/my_app/module_using_my_lib.ex:2: undefined function conn/0
    (elixir 1.10.3) src/elixir_locals.erl:114: anonymous fn/3 in :elixir_locals.ensure_no_undefined_local/3
    (stdlib 3.13) erl_eval.erl:680: :erl_eval.do_apply/6

I’ve tried wrapping conn in Macro.var, to no success. But to my understanding that shouldn’t even be necessary in this case.

4

Not sure what the forum policy on bumping is - if it is allowed consider this topic bumped. :slight_smile: If not I’ll go ahead and open a new topic for this second question.

5

Changing conn to var!(conn) will compile cleanly I believe.

1 Like
6

This seems to have done the trick, although I am not sure why this is required in this case. Thank you very much!

Now I am running into a different macro issue entirely:

My __using__ macro takes an opts argument which is a proplist with several options.

Since I want to post-process the options at compile time before inserting them into code I’ve gone ahead and implemented it in the following way (the implementation of normalize_module should not be relevant):

defmacro __using__(opts) do  
  quote do
    modules = unquote(opts[:modules])
    modules = modules |> Enum.map(&MyLib.normalize_module/1)
    
    def get_modules(), do: unquote(modules)
  end
end

This creates an issue in the unquote(modules) line, since the contents of modules is no longer coming from the macro, but its own body:

== Compilation error in file lib/myapp/module_using_my_lib.ex ==
** (CompileError) lib/myapp/module_using_my_lib.ex:2: undefined function modules/0
    (elixir 1.10.3) src/elixir_locals.erl:114: anonymous fn/3 in :elixir_locals.ensure_no_undefined_local/3
    (stdlib 3.13) erl_eval.erl:680: :erl_eval.do_apply/6
  • How would I go about inserting the current content of the “modules” variable in this line? The way I did it was the way one would approach it outside of a macro.
  • Is this the proper way to implement such a behavior? I also tried implementing it outside the quote part, which came with its own set of issues (including having to use Code.eval_quoted which I’d like to avoid).

Huge thanks to everyone who has been helping me out so far! :slight_smile:

7

Remember that everything in the quote block is actually runtime execution. Outside the quote is compile time execution. Therefore you are probably after:

defmacro __using__(opts) do  
  modules = 
    opts
    |> Keyword.get(:modules)
    |> Enum.map(&MyLib.normalize_module/1)

  quote do
    def get_modules(), do: unquote(modules)
  end
end

Do note however that macros both receive and return AST. So opts[:modules] may not be what you expect since it will be AST.

You may find that you need the following signature for MyLib.normalize_module/1:

def normalize_module({:__aliases__, _meta, [module_name]}) do
  module_name
end

or alternatively:

def normalize_module(module_ast) do
  Macro.expand(module_ast, __ENV__)
end
2 Likes
8

I see, thank you. That’s quite unfortunate, since this feels like it makes this far more difficult than it has to be.

For example my input AST can look like the following:

[
  {:%{}, [line: 10],
   [
     module: {:__aliases__, [counter: {MyApp.ModuleUsingMyLib, 1}, line: 10],
      [:MyApp, :Plug1]},
     some_opt: true,
     another_opt: {:%{}, [line: 10], [key: 1]}
   ]},
  {:__aliases__, [counter: {MyApp.ModuleUsingMyLib, 1}, line: 13],
   [:MyApp, :Plug2]}
]
# More variants are possible

Since the AST can be and will be deeply nested I’d be unable to use the “standard” elixir functions for interacting with data structures.
Also it feels wrong to make a function (the normalize_module function) dependent on receiving AST when it could potentially called at runtime with non-AST data.

Calling Macro.expand seems to do nothing with my AST.

Is there a better way to do this?

I’d ideally not receive this option as AST but as a “normal” data structure, which contents I can after modifying turn into a literal and insert into the code.

9

Since macros are primarily about manipulating code, not executing it, I’m not sure how your expectation could be met.

Nevertheless, you can evaluate AST at compile time with Code.eval_quoted/3 although as the documentation says:

Warning: Calling this function inside a macro is considered bad practice as
it will attempt to evaluate runtime values at compile time. Macro arguments are
typically transformed by unquoting them into the returned quoted expressions
(instead of evaluated).

Using your example:

defmacro __using__(opts) do  
  options = Code.eval_quoted(opts)
  modules = Keyword.get(options, :modules)

  quote do
    def get_modules(), do: unquote(modules)
  end
end

The benefit here isn’t really enough to warrant using Code.eval_quoted/3 so perhaps reverting closer to your original idea may be clearer since it too will get resolved at compile time (but in a later compiler expansion):

defmacro __using__(opts) do
  quote do
    modules = unquote(opts[:modules])
    @modules Enum.map(modules, &R.normalize_module/1)

    def get_modules(), do: @modules
  end
end
1 Like
10

I see, for some reason I always looked at them as a way of executing code at compile time, but this makes a lot more sense.

True, I never even considered the fact that a user might want to e.g. use a runtime configuration value. Then I guess I’ll budge and do the data transformation fully at runtime.

defmacro __using__(opts) do
   modules = opts[:modules]

   quote do
    def get_modules() do 
      unquote(modules)
      |> Enum.map(modules, &R.normalize_module/1)
    end
  end
end

I initially wanted to avoid this to save performance at runtime, but this doesn’t seem to be a good way to do it. Maybe this is a case of premature optimization being evil again :slight_smile:

Thanks for your time and patience. I wish I could mark the answers of multiple people as solution, but I’ll just go ahead and mark your last one.

Have a great weekend!

1 Like
11

In my somewhat lengthy reply you might have missed this version which is compile-time calculation of your modules:

12

Oh sorry, I didn’t check back!

I should have mentioned that in my reply: I did try this option, but it felt like a hack to use attributes for this. If I notice any performance problems I’ll probably go for this solution though.

Thanks again!