How to make a put_deep/2 macro (how to access a variable from a macro)?

Hello!

# Functions are easy!
m = %{}
put_deep(m, [:foo, :bar, :baz], 1)
%{foo: %{bar: %{baz: 1}}}

# Macros are hard.
m = %{}
put_deep(m.foo.bar.baz, 1)
# how??

Here’s my macro…

defmacro put_deep(path, value) do

  {_node, {var, keys}} = Macro.postwalk(path, nil, fn node, acc ->
    acc = case {node, acc} do
      {{var, _, nil}, nil} -> {var, []}
      {key, {var, keys}} when is_atom(key) -> {var, [key | keys]}
      {junk, acc} when is_tuple(junk) -> acc
    end
    {node, acc}
  end)

  keys = Enum.reverse(keys)
  IO.inspect(var)
  IO.inspect(keys)

  quote location: :keep do
    put_deep(unquote(var), unquote(keys), unquote(value))
  end
end

The IO.inspect() calls produces:

:m
[:foo, :bar, :baz]

So I have the variable name as a symbol, but I don’t know how to access that variable from the macro.

I’ve tried tons of combinations of Kernal.var!, Macro.var, and Macro.escape.

Thanks for the help!

I can’t help you with the macro magic. But do you know, that Kernel.put_in exists?

1 Like

Kernel.put_in both require all the keys to exist beforehand. put_deep creates the keys if they don’t exist.

m = %{}
put_deep(m, [:foo, :bar, :baz], 1)
%{foo: %{bar: %{baz: 1}}}

m = %{}
put_in(m, [:foo, :bar, :baz], 1)
** (ArgumentError) could not put/update key :bar on a nil value
1 Like

That’s not correct.

iex(1)> m = %{}
%{}
iex(2)> put_in(m, [Access.key(:foo, %{}), Access.key(:bar, %{}), Access.key(:baz)], 1)
%{foo: %{bar: %{baz: 1}}}
3 Likes

Besides what @LostKobrakai already said about put_in possibly already being useful as-is for you, it might still be nice to learn more about writing your own macros.

For an unhygienic macro like this, Kernel.var!(var) (which is a shorthand for Kernel.var!(var, nil)) should work to refer to the variable in the surrounding scope.

If this still does not work, could you showcase the exact code that you tried? Because then there probably is something else wrong with the code.

1 Like

Thank you for the correction.

Indeed that is the exact type of code we were trying to clean up though:

put_in(settings, Enum.map(["index", "mapping", "total_fields", "limit"], &Access.key(&1, %{})), field_count)

Here’s the module:

defmodule MapUtils do

  def put_deep(map, [key], value) do
    Map.put(map, key, value)
  end

  def put_deep(map, [key | path], value) do
    value = put_deep(map[key] || %{}, path, value)
    Map.put(map, key, value)
  end

  defmacro put_deep(path, value) do

    {_node, {var, keys}} = Macro.postwalk(path, nil, fn node, acc ->
      acc = case {node, acc} do
        {{var, _, nil}, nil} -> {var, []}
        {key, {var, keys}} when is_atom(key) -> {var, [key | keys]}
        {junk, acc} when is_tuple(junk) -> acc
      end
      {node, acc}
    end)

    keys = Enum.reverse(keys)
    |> IO.inspect()

    quote location: :keep do
      MapUtils.put_deep(var!(unquote(var)), unquote(keys), unquote(value))
    end
  end
end

And here’s example usage.

iex(3)> m = %{}
iex(4)> MapUtils.put_deep(m.foo.bar.baz, 123)
[:foo, :bar, :baz]
** (ArgumentError) expected a variable to be given to var!, got: :m
    (elixir 1.12.2) lib/kernel.ex:4300: Kernel."MACRO-var!"/3
    (elixir 1.12.2) expanding macro: Kernel.var!/1
    iex:4: (file)
    (aw 0.1.0) expanding macro: MapUtils.put_deep/2
    iex:4: (file)

Thanks for the help!

When working in macros its really helpful to quote in iex to see what the required AST is - especially when you’re splicing AST like this. The immediate issue is that you are treating the atom :m as a variable reference, but it isn’t.

iex> quote do: m     
{:m, [], Elixir}

So you can see that a variable reference requires a valid AST tuple be formed. For example:

  defmacro put_deep(path, value) do

    {_node, {var, keys}} = Macro.postwalk(path, nil, fn node, acc ->
      acc = case {node, acc} do
        {{var, _, nil}, nil} -> {var, []}
        {key, {var, keys}} when is_atom(key) -> {var, [key | keys]}
        {junk, acc} when is_tuple(junk) -> acc
      end
      {node, acc}
    end)

    keys = Enum.reverse(keys)
    |> IO.inspect()

    var = {var, [], Elixir}
    |> IO.inspect

    quote location: :keep do
      MapUtils.put_deep(var!(unquote(var)), unquote(keys), unquote(value))
    end
  end
end

Will work as you probably expected, Including that it will raise an exception if the variable m is not in scope in the calling context. I’d probably suggest using var = {var, [], __CALLER__.module} though which makes the scope clearer.

4 Likes

The next thing to consider might me “what if I just want to provide an empty map to insert the keys into?” The code above will fail because “%{}.foo.bar.baz” will not produce a variable reference, it will produce the AST for an empty map.

Here is an updated version which will allow either a map reference or a variable reference with an example module.

defmodule MapUtils do

  def put_deep(map, [key], value) do
    Map.put(map, key, value)
  end

  def put_deep(map, [key | path], value) do
    value = put_deep(map[key] || %{}, path, value)
    Map.put(map, key, value)
  end

  defmacro put_deep(path, value) do
    {_node, {var, keys}} = Macro.postwalk(path, nil, fn node, acc ->
      acc = case {node, acc} do
        {{var, _, nil}, nil} -> {var, []}
        # Additional clause to account for a non-variable
        # reference
        {var, nil} -> {var, []}
        {key, {var, keys}} when is_atom(key) -> {var, [key | keys]}
        {junk, acc} when is_tuple(junk) -> acc
      end
      {node, acc}
    end)

    keys = Enum.reverse(keys)

    case var do
      # Plan variable reference. Wrap is
      # as an AST tuple so it can be inserted
      var when is_atom(var) ->
        var = {var, [], __CALLER__.module}
        quote location: :keep do
          MapUtils.put_deep(var!(unquote(var)), unquote(keys), unquote(value))
        end

      # It's something else. Just insert the AST
      # as is.
      other ->
        quote location: :keep do
          MapUtils.put_deep(unquote(other), unquote(keys), unquote(value))
        end
    end

  end
end

defmodule MapUtils.Test do
  require MapUtils

  # The variable must be in scope at the
  # time of the macro call
  def test do
    m = %{}
    MapUtils.put_deep(m.foo.bar.baz, 123)
  end

  # Its not a variable reference so
  # the AST is inserted unmodified
  def test2 do
    MapUtils.put_deep(%{}.foo.bar.baz, 123)
  end
end
4 Likes

Lasy version of the macro, just to collapse the unneeded conditional logic (but it was easier to explain in that example):

  defmacro put_deep(path, value) do
    {_node, {var, keys}} = Macro.postwalk(path, nil, fn node, acc ->
      acc = case {node, acc} do
        # Add the var! reference here and now
        {{var, _, nil}, nil} -> {{:var!, [], [{var, [], __CALLER__.module}]}, []}
        {var, nil} -> {var, []}
        {key, {var, keys}} when is_atom(key) -> {var, [key | keys]}
        {junk, acc} when is_tuple(junk) -> acc
      end
      {node, acc}
    end)

    keys = Enum.reverse(keys)

    quote location: :keep do
      MapUtils.put_deep(unquote(var), unquote(keys), unquote(value))
    end
  end
2 Likes

Thank you so much for that detailed explanation!

I think I got close one time, but did this instead:

iex> var = Macro.var(:m, nil)
{:m, [], nil}

And I couldn’t discern from the docs what “context” should be.

You read my mind with “what about map literals”. Thanks again, that was super helpful… :slight_smile:

For posterity here is the final module.

I opted to use a recursive function to parse out the var/literal and keys from the AST instead of using Macro.postwalk/3. I think that greatly simplified the logic and increased readability.

Works for variables and map literals:

iex> m = %{}
%{}

iex> MapUtils.put_deep(m.foo.bar, 1)
%{foo: %{bar: 1}}

iex> MapUtils.put_deep(%{}.foo.bar, 1)
%{foo: %{bar: 1}}

iex> MapUtils.put_deep(%{baz: 2}.foo.bar, 1)
%{baz: 2, foo: %{bar: 1}}

iex> MapUtils.put_deep(m.foo["bar"].baz, 1)
%{foo: %{"bar" => %{baz: 1}}}

Module def:

defmodule MapUtils do

  def put_deep(map, [key], value) do
    Map.put(map, key, value)
  end

  def put_deep(map, [key | path], value) do
    value = put_deep(map[key] || %{}, path, value)
    Map.put(map, key, value)
  end

  defmacro put_deep(path, value) do
    {var, keys} = parse_path(path)

    var = if is_atom(var) do
      {:var!, [], [Macro.var(var, __CALLER__.module)]}
    else
      var
    end

    quote location: :keep do
      MapUtils.put_deep(unquote(var), unquote(keys), unquote(value))
    end
  end

  defp parse_path(ast, keys \\ []) do
    case ast do
      {{:., _, [Access, :get]}, _, [path, key]} -> parse_path(path, [key | keys])
      {:., _context, [path, key]} -> parse_path(path, [key | keys])
      {:%{}, _, _} = literal -> {literal, keys}
      {path, _, _} -> parse_path(path, keys)
      var when is_atom(var) -> {var, keys}
    end
  end

end

Thanks for all the help, been a great learning experience.

2 Likes