Macro expansion order

I’d like to understand macro expansion rules, so I created this piece of code:

lib/my_macro.ex:

defmodule MyMacro do
  defmacro print_module_attribute(attribute) do
    IO.inspect "ATTRIBUTE:"
    IO.inspect attribute
    IO.inspect "EXPANDED VALUE"
    IO.inspect Macro.expand attribute, __CALLER__
    quote do
      nil
    end
  end
end

lib/main.ex

defmodule Main do
  require MyMacro
  @my_attr :asdf
  MyMacro.print_module_attribute(@my_attr)
  def in_function do
    MyMacro.print_module_attribute(@my_attr)
  end
end

When running mix compile, I get the following output:

Compiling 2 files (.ex)
"ATTRIBUTE:"
{:@, [line: 4], [{:my_attr, [line: 4], nil}]}
"EXPANDED VALUE"
{:with, [],
 [{:<-, [],
   [{:when, [],
     [{{:_, [], Kernel}, {:doc, [counter: -576460752303423485], Kernel}},
      false]},
    {{:., [],
      [{:__aliases__, [alias: false, counter: -576460752303423485], [:Module]},
       :get_attribute]}, [],
     [{:__MODULE__, [counter: -576460752303423485], Kernel}, :my_attr,
      [{:{}, [], [Main, :__MODULE__, 0, [file: "lib/main.ex", line: 4]]}]]}]},
  [do: {:doc, [counter: -576460752303423485], Kernel}]]}
"ATTRIBUTE:"
{:@, [line: 6], [{:my_attr, [line: 6], nil}]}
"EXPANDED VALUE"
:asdf
Generated macro_problem app

It is a little bit surprising to me. When I call the macro in the module scope, it can’t expand the module attribute but when I call it from the function scope it can.

I wonder why is that? I am guessing that there are multiple expansion phases and compiler firstly expands the macro in module scope, secondly module attributes and thirdly function definitions.
However, defmodule and def are macros themselves so I am not sure if my theory makes sense.

Could you point me to a piece of documentation that explains it in details?

2 Likes

Module attributes are macros as well: https://hexdocs.pm/elixir/Kernel.html#@/1 and It seems one case does expand to what the @/1 macro expands to in terms of AST, while the other seems to expand to the result value. The difference might come from the difference between how defmodule handles it’s do: block vs. what the def macro does with it.

Can you elaborate on the difference between handling do: block in both? Is it documented somewhere or is it just a wild guess like mine? :slight_smile:

There are differences like this one, but I’m not sure if they are really related to what you’re seeing:

I believe the information you are missing is that def is a macro that stores some AST to be expanded and evaluated. So when a function body is expanded, the module body has been expanded and executed.

You can imagine your module, after expansion, becomes this:

defmodule Main do
  :elixir_internal.store_module_attribute(:my_attr, :asdf)
  MyMacro.print_module_attribute(:elixir_internal.read_module_attribute(:my_attr))
  :elixir_internal.store_function(:def, :in_function, quote do
    MyMacro.print_module_attribute(@my_attr)
  end)
end

Generally speaking, we should avoid doing work inside expansion.

And the reason why we delay expanding function bodies is because we don’t want to expand them if they are not meant to be compiled. Imagine you wrap your functions in an if block that does not evaluate to true. By postponing it, we avoid doing unnecessary work. The late expansion is also what allows dynamic definitions.

6 Likes

In another post, Jose Valim stated that all macros are expanded in order–before any executable code contained in a macro is executed. After all the macros have been expanded, then any executable code in each macro is executed. As a result, inside the macro expansions that follow the macro @my_attr :whatever, the value :whatever will not have been set yet (by the executable code contained in the @ macro), so the subsequent macro expansions use the value nil.

So when a function body is expanded, the module body has [already] been expanded and executed .

So the def macro is expanded at compile time, but the macros inside a def are not? Are you saying that the macros found inside a def are not expanded until the function is actually called at runtime?

In Metaprogramming Elixir on p. 13 it says:

When the compiler encounters a macro, it recursively expands it until the code no longer contains any macro calls.

The whole reason we programmers buy metaprogramming books is to learn the details not provided in the guides. In Metaprogramming Elixir, there is no mention of the delayed evaluation of macros inside a def. :frowning:

2 Likes

def macros need to expand at compile time. The compilation has a couple of phases and module attributes are available only after the compilation finished. If I understood Jose’s answer correctly, it works like this:

  1. Expand all macros in order recursively. In my example, it works like this:
  • require MyMacro makes sure that MyMacro module is already fully compiled
  • @my_attr: :asdf becomes “hey! register this attribute when evaluating final AST”
  • then expands the the macro MyMacro.print_module_attribute(@my_attr); at current point, attributes are not calculated (it prints “ATTRIBUTE: …” and "EXPANDED VALUE: "
  • then expands def macro but it is kind of special. It uses Macro.escape under the hood so it is not yet fully expanded. Again, it is something like register that AST under that function name
  1. Evaluate the module (it is not runtime!). Now we have the body so we can start evaluating:
  • calculate module attributes
  • evaluate first expanded macro (it evaluated to nil so nothing happens)
  • evaluate function definitions; Since they had the expansion delayed, we need to expand them now and that prints the second “ATTRIBUTE: …” and “EXPANDED VALUE: …”. Attributes were already evaluated.

I hope I get it right now :slight_smile:

I’d love to see this mechanism explained in greater detail in “Metaprogramming Elixir”!

2 Likes

Yes! And another thing the book omits: ALL the differences between bind_quoted() and unquote(), which is something that really tripped me up. For instance, bind_quoted() does NOT make the specified variables available inside a def in the quote block, where unquote() does. I don’t think I’ll ever use bind_quoted() because of that feature. The book makes it seem like bind_quoted() is a drop in replacement for unquote(), whereas, as far as I can tell, bind_quoted() is merely equivalent to:

defmacro two(x) do
  quote do
    binded_x = unquote(x)  #You get one evaluation of x like with bind_quoted().

    #Now use binded_x where it can be seen, which is not inside a def, like calc():
    IO.puts "The macro arg in two() was: #{binded_x}."

    def calc do
      IO.puts "The macro arg in two() was:  Can't see it!"
    end

  end
end

I see no reason to use bind_quoted() when it can cause so much pain when used as a drop in replacement for unquote(). I’ll just use the longcut above, so that I’ll know what’s going on.

The problem here is not that bind_quoted is not working, but that unquote fragments within macros quote do need „two levels“ of unquoting (outside quote -> inside quote and inside quote -> inside function). That‘s exactly the place you even need bind_quoted to resolve the ambiguity of which task unquote is fulfilling.

I believe @sasajuric talks about this in his series but it has been a while since I last read it: https://www.theerlangelist.com/article/macros_1

bind_quoted actually disables unquoting for the current quote, this is mentioned in the docs. There are actual use cases for it, such as handling two level of unquotes, as mentioned by @LostKobrakai.

I’m not sure what you are trying to say there, but I don’t need two levels of unquoting here:

defmodule My do

  defmacro one(x) do
    quote do
      def show_in_one do
        IO.puts "The macro arg for one() was: #{unquote(x)}"
      end
    end

  end
end

I meant something like this, but actually didn’t know that a simple unquote in a hardcoded functions body would still work.

defmodule A do
    defmacro gen_funcs(list) do
        quote bind_quoted: [list: list] do
            # The following instead of bind_quoted wouldn't work
            # list = unquote(list)
            for {name, i} <- Enum.with_index(list, 1) do
                def unquote(name)(), do: unquote(i)
            end
        end
    end
end
    
defmodule B do
    require A
    A.gen_funcs([:test, :unquote, :fragments])
end
    
IO.inspect(B.test()) # 1
IO.inspect(B.unquote()) # 2
IO.inspect(B.fragments()) # 3
1 Like

Like I said:

bind_quoted() does NOT make the specified variables available inside a def in the quote block, where unquote() does.

The issue is that when you do that, you are changing the semantics of the code:

Imagine if someone use your one macro like this:

defmodule Using do
  import My
  x = 1
  one(x)
end

In your version, it is not going to work, because you are taking the variable x out of its context and putting it inside a function. The correct is to use bind_quoted and use the two-level unquoting:

defmodule My do
  defmacro one(x) do
    quote bind_quoted: [x: x] do
      def show_in_one do
        IO.puts "The macro arg for one() was: #{unquote(x)}"
      end
    end
  end
end

EDIT: of course, the above depends if your macro wants to work with values or expressions but most macros (think Phoenix, Ecto, etc) do work on values.

2 Likes

I explained some simplified version in part 1 (“Compilation process”), and part 6 (“Order of expansion” and “Module-level friendly macros”). This was mostly based on my own intuition, some experiments, and bit of code analysis.

2 Likes