Accessing macro/caller context variables in dynamically generated of functions in macros

Questions / Help
,
1

I am trying to understand how macros work – specifically how macro/caller contexts work. I stumbled upon these set of codes that have differing behaviors which conflict with my current understanding of these contexts.

(1) In the first code example below, I am attempting to access a caller context variable from a dynamically generated function.

defmacro foo do
  quote do
    x = 1 + 1
    def execute(), do: IO.inspect(unquote(x))
  end
end

However, this does not work. It presents the following error:

** (CompileError) ambiguous.exs:14: undefined function x/0
    (elixir 1.11.3) src/elixir_locals.erl:114: anonymous fn/3 in :elixir_locals.ensure_no_undefined_local/3
    (stdlib 3.15) erl_eval.erl:685: :erl_eval.do_apply/6

(2) In the second code example, I attempt to do the same thing, except the caller context variable is retrieved differently.

defmacro foo(opts) do
  quote bind_quoted: [opts: opts] do
    x = Keyword.get(opts, :x)
    def execute(), do: IO.inspect(unquote(x))
  end
end

I can actually access the caller context variable x from the dynamically generated function.

From what I can gather, the two code examples above are of similar nature – even if the way the results are reached. Both include the dynamically generated function (execute) attempting to access a caller context variable (x) through unquote. The former assigns this variable through a simple expression while the retrieves this from a keyword list that has been bound to the caller context through bind_quoted.

(3) Even more interestingly, the following code example works:

defmacro foo(x) do
  quote do
    def execute(), do: IO.inspect(unquote(x))
  end
end

Where the dynamically generated function is able to access a macro context variable.

My question is how do the examples differ, especially the first two code examples? What exactly is happening that allows the second to work but not the first? Why does the third work but the first doesn’t?

Note: I am using this macro in the same way across (1), (2), and (3):

defmodule Applied do
  require Confusion # this is the module that contains the macro food
  Confusion.foo() # for (1)
  Confusion.foo(x: 5) # for (2)
  Confusion.foo(5) # for (3)
end

Any help would be great! Thanks!

2

The difference is using the bind_quoted option, which implicitly sets the unquote: false option. Setting the quote call with unquote: false will enable the contents of the do block to use unquote fragments, which is what your second example uses. Not setting that means the unquote references variables in the macro‘s scope. Your first example fails, because there is no x in scope and your third example succeeds.

4 Likes
3

Thank you for your response! @LostKobrakai

Not setting that means the unquote references variables in the macro‘s scope.

I’m assuming this is so because the default behavior of unquote in the caller’s context is to attempt to inject an AST expression from the macro’s context into the caller’s context? This extends to dynamically generated functions?

Would this be a reference to this section in the documentation here?

defmacro defkv(kv) do
  quote do
    Enum.each(unquote(kv), fn {k, v} ->
      def unquote(k)(), do: unquote(v)
    end)
  end
end

My explanation for this would be:

unquote(k) is ambiguous because it could be an unquote fragment or a regular unquote. This ambiguity is caused because the unquote attempts to look to the macro’s context for the variable k instead of the caller’s context. This is because when the macro is injected into the caller’s call site, unquote(k) could be referencing an injected value k (i.e. an unquote fragment) but since the default behavior of unquote is to reference a macro context variable, unquote(k) could be referencing a macro context variable, making it a regular unquote.

As such, by doing something like:

defmacro defkv(kv) do
  quote bind_quoted: [kv: kv] do
    Enum.each(kv, fn {k, v} ->
      def unquote(k)(), do: unquote(v)
    end)
  end
end

We are disabling unquote which effectively disables the default behavior of looking to the macro’s context for variables. So the only scope that’s available is the caller’s context. Thus, we inform the compiler that unquote(k) can only look to the caller’s context for a variable k and so the code above works as intended? We have removed the ambiguity of whether the variable belongs to macro’s context or caller’s context as we basically guarantee that the variable can only appear in the caller’s context.

So back to my original problem:

(1) does not work as unquote(x) attempts to reference a macro context variable which does not exist. Disabling unquote informs the compiler to look at the caller context only for x which makes it work as intended

(2) works because bind_quoted already disabled unquote so there is no ambiguity to which scope x belongs to as it will be the caller’s context by default

(3) works because we intend to reference the macro context variable x so unquote(x) has no ambiguity whatsoever

Would this be an accurate explanation of what’s going on?

Thank you so much!

4

To maybe simplify a bit its important to remember what a macro is:

  1. It is a function whose arguments are AST and which is required to return AST. Other than the very special case of the __CALLER__ special form, there is not really a “caller context/callee context”. There is just AST -> AST.

  2. The compiler expands macros at compile time and inserts the AST returned from the macro into the call site of the macro. Then compilation continues to proceed (recursively expanding macros before moving on to code generation).

  3. defmodule , def , defp , defmacro , and defmacrop are all macros. This is relevant when considering unquote fragments later on.

With that in mind, back to your original questions:

defmacro foo do
  quote do
    x = 1 + 1
    def execute(), do: IO.inspect(unquote(x))
  end
end

Produces a compiler error because when the compiler is expanding unquote(x) there is no variable x in scope. The x = 1 + 1 is just AST in a quote block.

defmacro foo(opts) do
  quote bind_quoted: [opts: opts] do
    x = Keyword.get(opts, :x)
    def execute(), do: IO.inspect(unquote(x))
  end
end

Works because as @LostKobrakai says, bind_quoted also sets unquote: false. The net affect of this is that the unquote(x) expression is not unquoted inside the macro foo. The entire AST - including the unquote(x) is inserted into the calling site. It is expanded then when the calling code is further expanded and therefore unquote(x) is expanded in a later phase and at that phase x is in scope because of the preceding x = Keyword.get(opts, :x). Note too that the code that is inserted in the calling site will look like x = Keyword.get([x: "option x"], :x) because of the bind_quoted: [opts: opts].

BTW, the reason inserting code that includes unquote(x) into the calling site works is because def is also a macro. And therefore unquote can be expanded in the context of the def. Hence unquote fragment, But this is not about caller/callee context. It is just about when a macro is expanded. unquote: false basically allows you to manage nested quote blocks and to define whether the unquote is expanded now or later.

defmacro foo(x) do
  quote do
    def execute(), do: IO.inspect(unquote(x))
  end
end

By the same rules, unquote(x) works because x is in scope at the time of macro expansion (before its inserted into the call site) because x is outside the quote block.

4 Likes
5

Thank you for your response! @kip

Okay so let’s say we stop looking at this problem from the perspective of macro/caller context scoping. Rather, we look at the availability of variables during macro expansion (i.e. when macros are expanded) and how this interacts with unquote.

Going back to the example given by the documentation:

defmacro defkv(kv) do
  quote do
    Enum.each(unquote(kv), fn {k, v} ->
      def unquote(k)(), do: unquote(v)
    end)
  end
end

Even before macro expansion, unquote(k) is attempted by the compiler as unquotes are not disabled. This means that it attempts to inject the AST of a variable k that’s not yet in scope since it has to be injected first. k is only in scope during macro expansion. So by disabling unquote from happening before macro expansion, it defers the unquoting (since def is also a macro) to be performed during macro expansion (or any later phase), when k is finally in scope.

The ambiguity between unquote fragment or regular unquote arises from the injected form of the macro; which is similar to the code example provided in the documentation. The code example is the “ideal” scenario.

kv = [foo: 1, bar: 2]
Enum.each(kv, fn {k, v} ->
  def unquote(k)(), do: unquote(v)
end)

This is the code that we want our macro to produce and it relies on unquote fragments. But by allowing unquote to attempt to inject the AST of k before macro expansion, we are no longer relying on unquote fragments. Rather, we are relying on regular unquotes, which has the problems raised above.

defmacro defkv(kv) do
  quote bind_quoted: [kv: kv] do
    Enum.each(kv, fn {k, v} ->
      def unquote(k)(), do: unquote(v)
    end)
  end
end

On the other hand, since this code preserves unquote as is without attempting to evaluate it before macro expansion, it is able to be evaluated during the expansion of def which allows the injected code to be similar to the unquote fragment we wanted. This also means that any variables we declare in quote is in scope already.

So in my original problem,

(1) does not work since x is not yet in scope when we are attempting to unquote it. Disabling unquote during the initial stages of parsing allows x to first be injected into scope before we attempt to unquote it allows us to access this variable and thus, allowing the code to work

(2) works because bind_quoted disables unquotes already so the above logic applies here

(3) works because x would have already been available during macro expansion since x is not in the quote block, which means it does not need to be expanded before it’s in scope

Does this explanation properly capture the essence of what you were getting at? :open_mouth:

Thank you!

6

Very close!

Instead of:

On the other hand, since this code preserves unquote as is without attempting to evaluate it before macro expansion

I would say:

On the other hand, since this code preserves unquote as is without attempting to evaluate it before this macro expansion (unquote still has to be evaluated, just not right now).

You might find the docs for:

The key being that macro expansion is a recursive process, it keeps going until there is nothing to expand.

2 Likes
7

I see, so it was a misunderstanding of how macros expand that was my issue! Thank you so much for helping to clear this up! @kip @LostKobrakai