Dynamically create a map inside a function declared in a macro quote block

Hi,

I’ve a little macro that given some attributes in the form keys / getters like:

defmodule Foo do
  use Foo.Macro

  # simplified version, normally there's a little DSL that defines the following map
  event :something do
    %{
      foo: {Map, :get, [:somekey]},
      bar: {Map, :get, [:someotherkey]}
    }
  end
end

adds a parse(data) function to the Foo module which returns a map with the above keys and values extracted from data using the given accessor.

The macro is something like:

defmodule Foo.Macro do
  defmacro __using__(_opts) do
    quote do
      import Foo.Macro
    end
  end

  defmacro event(_name, do: block) do
    quote do
      def parse(event) do
        unquote(block)
        |> Enum.map(fn {key, parser} ->
          {m, f, args} = parser
          {key, apply(m, f, [event | args])}
        end)
        |> Map.new()
      end
    end
  end
end

Which works without problems.

The question now is: is possible in someway to dynamically create a function in the form:

def parse(event) do 
  %{
    bar: Map.get(event, someotherkey),
    foo: Map.get(event, somekey)
  }
end

Without having to resort to a runtime Enum.map, since all map keys and accessor functions/args are known at compile time ?

I’ve tried all sort of stuff, like composing a string of code, quoting it, etc etc but no result till now. Wondering if this is possible in Elixir (I’m not a super macro expert at all…)

In the real world this is a sort of map translator, where the data has somewhat many keys and accessor functions can be much more complex but always needs the whole data as input, since the value returned for field X may be dependent also on other fields.

Ready to learn the best way to do that, if any! :smiley:

PS: the relevant files are also in elixir-macro-experiment/lib at main · xadhoom/elixir-macro-experiment · GitHub

Well, after fiddling, here what seems to work, but is really really really hacky / ugly:

defmodule Foo.Macro do
  defmacro __using__(_opts) do
    quote do
      import Foo.Macro

      @before_compile {Foo.Macro, :inject_ast}
    end
  end

  defmacro event(_name, do: block) do
    quote do
      ast =
        unquote(block)
        |> Enum.reduce("def parse(event), do: %{", fn {key, parser}, acc ->
          {m, f, args} = parser

          args =
            ["event" | args]
            |> Enum.map(fn
              arg when is_binary(arg) -> arg
              arg -> "#{inspect(arg)}"
            end)

          ~s"#{acc} #{key}: #{inspect(m)}.#{f}(#{Enum.join(args, ", ")}),"
        end)
        |> Kernel.<>("}")
        |> Code.string_to_quoted!()

      Module.put_attribute(__MODULE__, :ast, ast)
    end
  end

  defmacro inject_ast(_env) do
    Module.get_attribute(__CALLER__.module, :ast)
  end
end

Basically, I’m building my translator fun as a string, converting to AST and then inject the AST into the module leveraging the @before_compile hook. Is this the correct way to do it? Having to build the code as a string seems an hack to me, for sure there’re should be a better way…

Definitely some room for improvement - and I’m happy to help but I’m confused on what this looks like in practise. Your use of do block is not so common to me, and I can’t tell from your example whether the code in the do block can be resolved at compile time or whether it must be resolved at runtime.

Can you show an example of how you would call this macro in practise?

1 Like

It’s meant to be resolved at compile time.

Sure!

Given the module

defmodule Foo do
  use Foo.Macro

  event :something do
    %{
      foo: {Map, :get, [:somekey]},
      bar: {Map, :get, [:someotherkey]}
    }
  end
end

After expansion, I can call Foo.parse(%{someotherkey: "hello"}) which results in %{foo: nil, bar: "hello"}

Here’s something that might help you. Its not exactly the transform you’re after because frankly I can’t get my head around the mental model of what you’re building. However it should get you on the right track:

defmodule Foo.Macro do
  defmacro event(_name, do: events) do
    quote bind_quoted: [events: events] do
      for {key, {module, function, [arg]}} <- events do
        def parse(%{unquote(arg) => value}) do
          %{unquote(key) => value}
        end
      end
    end
  end
end

defmodule Foo.Call do
  import Foo.Macro

  event :something do
    %{
      foo: {Map, :get, [:somekey]},
      bar: {Map, :get, [:someotherkey]}
    }
  end
end

In this way you’re defining N parse functions that extracts a single value, not a single function that given a map returns a map with new keys (as defined in the event macro) and values extracted from the original data using an “extractor” function.

Let’s see in this way: given input data

data = %{foo: "bar", baz: "what"}

and “schema” (the one passed to the event macro)

%{
  anotherfoo: {Map, :get, [:foo]},
  anotherbar: {Map, :get, [:baz]}
}

I can issue a single call Foo.parse(data) which returns (given the above schema) %{anotherfoo: "bar", anotherbar: "what"}. This tranformation is made using the passed fun, Map.get/2 in this example.

Basically I’m building a translator that given some data in map format, extracts some values from it and assign them to new keys (call it object normalization or data transformation, whatever)

You could define your event macro like this:

defmacro event(_name, do: block) do
  {:%{}, _, kw} = block

  quote do
    def parse(event) do
      unquote(
        {:%{}, [],
         kw
         |> Enum.map(fn {key, {:{}, _, [mod, f, args]}} ->
           {key,
            quote do
              apply(unquote(mod), unquote(f), [event | unquote(args)])
            end}
         end)}
      )
    end
  end
end

This works for your given example (simplified) use case. Note that this requires a map literal, as in your example, as the argument to event, as opposed to any expression whose value is a map, as it is directly transforming the AST of the given map.

To avoid this limitation you could, in theory, replace the first line of the macro definition with:

{map, _} = Code.eval_quoted(block)
{:%{}, _, kw} = Macro.escape(map)

However, the docs for Code.eval_quoted warn that using it inside a macro in this way is considered bad practice, and I’d agree with that assessment. I’d suggest trying to avoid doing this way, even if it means needing to rethink the way your DSL works.

1 Like

I agree with @zzq. Try not to use Code.eval_quoted or the @before_compile special attribute if unnecessary.

You can do it a bit more beautifully like this:

defmacro event(_name, do: block) do
  {:%{}, _map_meta, keys_values} = block

  keys_values =
    keys_values
    |> Enum.map(fn {key, {:{}, meta, [mod, fun, args]}} ->
      call =
        quote do
          unquote(mod).unquote(fun)(
            unquote_splicing([Macro.var(:event, __MODULE__) | args])
          )
        end

      {key, call}
    end)

  quote do
    def parse(event) do
      %{
        unquote_splicing(keys_values)
      }
    end
  end
end

I don’t quite see if your DSL is really going to help you. Maybe it would be better to keep it more simple and use a function that simply renames map keys according to another map (check out Map.map/2). Perhaps you can define your requirements with a bit more detail so we can help you better.

First of all, thanks for the responses, I had not digged into building the AST in this way and is great :slight_smile:

Well, I kept the example intentionally simple, in the real world there’s a little more DSL like:

defmodule Foo.Schema.UserCreate do
  @moduledoc """
  Schema and mapper for UserCreate event
  """
  use Foo.Schema

  alias Foo.Definitions.{Contact, Address}

  event UserCreate do
    definition(Contact)
    definition(Address)
  end
end

Where the definitions modules are something like:

defmodule Foo.Definitions.Contact do
  @moduledoc """
  Definition for contact data
  """
  use Foo.Schema.Definitions

  field(:name, one_of(["name", "given_name"]))
  field(:surname, one_of(["surname", "family_name"]))
  field(:age, const("age"))
end

So definitions modules define ad accessor function (like Map.get/2), and the UserCreate module is composed by definition modules. Right now I’m doing something like:

  • each “definition” module like Foo.Definitions.Contact stores fields and accessor funs (like one_of) in the module attributes thanks to the module Foo.Schema.Definitions and the field macro
  • each “schema” module like Foo.Schema.UserCreate retrieves accessor functions from each “definition” module and creates the parse function we’re discussing about.

So at the end of the game, a module in the form

defmodule Foo.Schema.UserCreate do
  @moduledoc """
  Schema and mapper for UserCreate event
  """
  use Foo.Schema

  alias Foo.Definitions.{Contact, Address}

  event UserCreate do
    definition(Contact)
    definition(Address)
  end
end

should be added a function like:

def parse(event) do
  %{
    name: Foo.Definitions.oneof(event, ["name", "given_name"]),
    surname: Foo.Definitions.oneof(event, ["surname", "family_name"]),
    age: Foo.Definitions.const(event, "age")
    ...
  }
end

Right now I’m not using any Code.eval_quoted but only the @before_compile hook and Code.string_to_quoted, like said above, which is more or less similar to the given answers except that I’m building the whole function as an ast instead of only the function body (mainly because fiddling with module attributes is simply not possible outside the quote block).

Just for fun I’ve benchmarked the map built into the fun against the runtime Enum.map/2 solution and is 2x faster, using 6x less memory (for a data payload with 100 elements and a schema with 30 fields). In our use case this is good since we have a fast stream of events that cannot be parallelized, so need to parse them as fast as possibile.

Thanks for providing more info! I’ll try to answer again later.

Just a quick note regarding my code, specifically: Macro.var(:event, __MODULE__)
It should actually be Elixir or __CALLER__.module instead of __MODULE__, but then it wouldn’t work and I couldn’t figure out why the compiler was saying that event is undefined (actual msg: undefined function event/0). Maybe someone has an idea?

hi @xadhoom , maybe i wrong but this can help:

defmodule Foo.Macro do
  defmacro __using__(_opts) do
    quote do
      import Foo.Macro
    end
  end

  defmacro event(_name, do: block) do
    quote do
      def parse(event) do
        var!(event) = event
        
        unquote(block)
      end
    end
  end
end


defmodule Bar do
  use Foo.Macro

  event :something do
    %{
      bar: Map.get(event, :bob)
    }
  end
end

defmodule Bob do
  use Foo.Macro

  event :something do
    %{
      bob: Map.get(event, :foo)
    }
  end
end


IO.inspect Bar.parse(%{bob: :buu})
IO.inspect Bar.parse(%{bob: :bipp})
IO.inspect Bob.parse(%{foo: :juu})

prints

%{bar: :buu}
%{bar: :bipp}
%{bob: :juu}

Its a small thing, but using __using__/1 to only import macros seems unnecessary obfuscation. As the Elixir documentation says:

Therefore use this function with care and only if strictly required. Don’t use use where an import or alias would do.

Okay, now I can see better what you’re trying to achieve. Here are some further suggestions:
If definition is a macro (defined in Foo.Schema?) that should expand to a map (e.g. %{name: oneof(...)) then you have to evaluate it, in order to get the resulting AST, otherwise you’ll only have something like {:definition, [], [{:__aliases__, [alias: false], [:Contact]}]}. You can do that by using this common pattern:

# Pass __ENV__ as `env`.
def get_definition_map(expr, env) do
  case expr do
    {:%{}, _map_meta, _keys_values} = map ->
      map

    expr ->
      case Macro.expand_once(expr, env) do
        ^expr ->
          raise ArgumentError, "unknown expression #{inspect(expr)}"

        expanded ->
          get_definition_map(expanded, env)
      end
  end
end

So you’ll use get_definition_map/2 on each expression in the block macro parameter (it may be a single expression or a list). It should be simple to reduce all the “definition maps” to a single keyword list. Finally pass that to the Enum.map(fn {key, ...} -> ... end) function similar to my code above. You need to work out how to match and convert those oneof and const functions, of course.

Maybe this is not the solution but I hope I was able to bring you closer to it.