Getting variable by its dynamic atom name from the caller context in a macro

Questions / Help
, ,
1

I’m struggling with overriding hygiene during manual manipulation of AST with Macro.postwalk. In short, I want to provide a syntax for interpolating variables with ^ in a DSL, similar to Ecto. So this:

some_col_name = "foo"
...
select [:x, ^some_col_name]

should be converted internally to

%Select{
  columns: ["x", "foo"]
}

There is more logic around that, though, so I need to first replace these ^some_col_name in the AST with the values of corresponding variables, and then do some more manipulation with the AST.

My attempts so far were variations on a theme:

Macro.postwalk(quoted, fn
  {:^, [], [{v, [], _}]} when is_atom(v) ->
    quote do
      var!(unquote(v))
    end
  x -> x
end)

I know that I need to override hygiene, but it seems var! only works with literal variable names, and there’s no example of using dynamic atom names with it that I could find. I also tried different combinations of Macro.var, unquote, and var!, to no avail.

Any suggestions?

1 Like
2

var! is a macro, so code like this:

var!(some_variable_name)

calls var! with {:some_variable_name, metadata, context}.

I found this example helpful:

defmodule MacroExample do
  defmacro get_value(variable) do
    IO.inspect(variable)
    quote do
      var!(unquote(variable))
    end
  end
end

iex(25)> require MacroExample
MacroExample
iex(26)> target_var = 42
42
iex(27)> MacroExample.get_value(target_var)
{:target_var, [line: 27], nil}
42

In your dynamic case, you likely want to capture the whole tuple and unquote it:

Macro.postwalk(quoted, fn
  {:^, [], [{v, _, _} = v_tuple]} when is_atom(v) ->
    quote do
      var!(unquote(v_tuple))
    end
  x -> x
end)
1 Like
3

Thanks for the help! However, that’s not the whole story, it seems :slight_smile: Let’s look at a more complete example:

defmodule Foo do
  defmodule Column do
    defstruct [:name]
  end 

  defmacro select(columns) do
    interpolate(columns)
    |> Enum.map(&atom_to_column/1)
    |> Macro.escape()
  end 

  def interpolate(q) do
    IO.inspect(q, label: :q)

    ast = Macro.postwalk(q, fn
      {:^, _, [{name, _, _} = v]} when is_atom(name) ->
        quote do
          var!(unquote(v))
        end
      x -> x
    end)

    IO.inspect(ast, label: :ast)

    ast 
  end 

  def atom_to_column(a) when is_atom(a) do
    %Column{name: a}
  end 
end

Here we define a Column struct, and a macro select that accepts a list of of terms, and converts those terms to Columns. We can only convert atoms to Columns, but we hope to handle interpolated variables (also containing atoms, for example) with the help of interpolate.

Plain atom list works as expected:

iex> import Foo
iex> select([:foo, :bar])
q: [:foo, :bar]
ast: [:foo, :bar]
[%Foo.Column{name: :foo}, %Foo.Column{name: :bar}]

Now let’s try the variable:

iex> bar = :foobar
iex> select([:foo, ^bar])
q: [:foo, {:^, [line: 50], [{:bar, [line: 50], nil}]}]
ast: [:foo, {:var!, [context: Foo, import: Kernel], [{:bar, [line: 50], nil}]}]
** (FunctionClauseError) no function clause matching in Foo.atom_to_column/1    
    
    The following arguments were given to Foo.atom_to_column/1:
    
        # 1
        {:var!, [context: Foo, import: Kernel], [{:bar, [line: 50], nil}]}
    
    iex:70: Foo.atom_to_column/1
    (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
    (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
    expanding macro: Foo.select/1 
    iex:50: (file)

I understand that the problem is because we are essentially replacing ^bar with var!({:bar, [], nil}), but what I want to do is to replace it with the value of bar, going from ^bar to :foobar in the AST output of interpolate. I wonder if that is even possible?

4

The REPL is obscuring things somewhat - interpolate runs during compilation and creates an AST which is then evaluated. bar has a value during the second part of that process.

Here’s a revised version that handles the example from your post:

defmodule Foo do
  defmodule Column do
    defstruct [:name]
  end

  defmacro select(columns) do
    interpolate(columns)
  end

  def interpolate(q) do
    IO.inspect(q, label: :q)

    ast = Macro.postwalk(q, fn
      {:^, _, [{name, _, _} = v]} when is_atom(name) ->
        quote do
          atom_to_column(var!(unquote(v)))
        end
      a when is_atom(a) ->
        quote do
          atom_to_column(unquote(a))
        end
      x -> x
    end)

    IO.inspect(ast, label: :ast)

    ast
  end

  def atom_to_column(a) when is_atom(a) do
    %Column{name: a}
  end
end

Running this produces output like:

iex(32)> select([:foo, :bar])
q: [:foo, :bar]
ast: [
  {:atom_to_column, [context: Foo, import: Foo], [:foo]},
  {:atom_to_column, [context: Foo, import: Foo], [:bar]}
]
[%Foo.Column{name: :foo}, %Foo.Column{name: :bar}]
iex(33)> wat = :foobar
:foobar
iex(34)> select([:foo, ^wat])
q: [:foo, {:^, [line: 34], [{:wat, [line: 34], nil}]}]
ast: [
  {:atom_to_column, [context: Foo, import: Foo], [:foo]},
  {:atom_to_column, [context: Foo, import: Foo],
   [{:var!, [context: Foo, import: Kernel], [{:wat, [line: 34], nil}]}]}
]
[%Foo.Column{name: :foo}, %Foo.Column{name: :foobar}]
5

After more trial and error I arrived at an alternative version, which seems even simpler:

defmodule Foo do
  defmacro select(columns) when is_list(columns) do
    # in macro context we replace all `^foo` forms with `foo`
    columns = Macro.postwalk(columns, &interpolate/1)
    IO.inspect(columns, label: :columns_after_interpolate)

    # we unquote in the caller context, and map atoms to Columns after all
    # `^foo` were replaced with `foo`
    quote do
      Enum.map(unquote(columns), &Foo.atom_to_column/1)
    end
  end

  def interpolate({:^, _, [{name, _ctx, _env} = v]}) when is_atom(name) do
    IO.inspect(v, label: :replaced_in_function)
    v 
  end
  def interpolate(x), do: x

  # can be extracted to a module!

  defmodule Column do
    defstruct [:name]
  end

  def atom_to_column(a) when is_atom(a) do
    %Column{name: a}
  end

  def atom_to_column(a), do: a
end

And it works:

iex> c = select([:foo, :bar])

columns_after_interpolate: [:foo, :bar]

[%Foo.Column{name: :foo}, %Foo.Column{name: :bar}]

iex> bar = :foobar 
iex> c = select([:foo, ^bar])

replaced_in_function: {:bar, [line: 458], nil}
columns_after_interpolate: [:foo, {:bar, [line: 458], nil}]

[%Foo.Column{name: :foo}, %Foo.Column{name: :foobar}]

Thank you so much for the help!

6

The alternative version works in the REPL, but so would a plain function (without the ^ notation). The usual reason for choosing a macro over a function would be to do things with ASTs that can’t be done at runtime. For instance:

case something do
  select([:foo, :bar]) -> IO.puts("OH HAI")
end

will fail to compile with invalid pattern in match, & is not allowed in matches.

Here’s another implementation that works a little bit differently; you’d write your example as select([:foo, bar]) without the ^:

defmodule Foo do
  defmodule Column do
    defstruct [:name]
  end

  defmacro select(columns) do
    Enum.map(columns, &interpolate/1)
    |> IO.inspect(label: :ast)
  end

  def interpolate(q) do
    IO.inspect(q, label: :q)

    quote do
      %Column{name: unquote(q)}
    end
  end
end

Since this constructs ASTs, it can be used in places where a plain function couldn’t:

select([:foo, new_var]) = select([:foo, :bar])
# new_var now has the value :bar 

case select([:foo, :bar]) do
  select([first_thing, second_thing]) -> {first_thing, second_thing}
end
# evaluates to {:foo, :bar}

fixed_thing = :foo 
case select([:foo, :bar]) do
  select([^fixed_thing, second_thing]) -> second_thing
  _ -> "nope"
end
# evaluates to :bar

One question your original post didn’t answer: what are you planning to make plain variables (no ^) do when passed to select? Ecto.Query repurposes them to be table references.

7

Thanks for the additional info!

There’s a bunch of other transformations that happen with those macros, for example this:

select [avg(to_number(:x)), "foo"]

will be ultimately converted to something similar to:

%Select{
  columns: [
    %Column{
      function: :avg,
      arguments: [
        %Column{function: :to_number, name: :x}
      ]
    },
    %Column{ 
      value: "bar",
      value_type: "string"
    }
  ]
}

avg and to_number don’t actually exist as a function, so using a macro is required. This is all a part of a larger framework, where you can create custom queries with syntax like:

query do
  select [avg(to_number(:x)), "foo", ^some_var]
  from "some_source_name"
  group_by [:something_else]
  having :x > 100.0
end

The ability to inline those ^some_var expressions is the only thing missing for the main functionality. I was hopeful that I can isolate them into a separate function, and just operate on a AST where variables are fully replace with their values, thus simplifying other transformations (since they don’t need to care about handling variables anymore). Also note, that each of the clauses should work individually, and both in modules and in REPL.

However, after more trials, investigation, and reading, I believe that this is probably not feasible - right now the whole code generation is happening at compile-time, but if I were to attempt to pre-process variable bindings beforehand, I would necessarily need to return to the caller context before continuing with my transformations.

I think it can be theoretically achivied by, for example, generating a lambda that calls itself in the caller, with a call to a macro that does post-processing; or by injecting code that uses an Agent to maintain runtime state (as in Metaprogramming Elixir’s example with HTML-DSL). But that is more complicated and harder to maintain, then handling variables in each clause, so I decided to revert back to my original solution.

The distinction between caller and macro contexts is something that I found myself rather confused about sometimes - haven’t done much metaprogramming before switching to Elixir a couple of years back, I guess that requires a bit more practice :sweat_smile:

P.S. Also, I played with Macro.expanding Ecto queries today, and apparently they also just inline those variables as variables AST in the end, so that everything is replaced only after returning to the caller context.

Also, I’m not so sure I will keep the ^ notation. Probably just using plain variables will be sufficient for my use-case :slight_smile: