Elixir macros - easy to understand, painful to implement

Hey guys - been hacking on Elixir projects for about two years now. I implemented a simple Instagram API wrapper library (exstagram) as well as an options pricing library. I’m working on an API wrapper client for Deribit, and because so many of the API endpoints start with “get” I figured it would be a good application of macros.

I learned a lot but had some trouble and wanted to ask some follow-up questions about what I could do better.
For reproduction/viewing my code, please see https://github.com/arthurcolle/Deribit.ex (feel free to git clone, etc).

Under lib/deribit/api/ you can see public.ex, which calls the macro - Macros.generate_endpoint() which accepts one atom as an argument (for now… adding query param/optional params support later). This has an atom passed in, which creates the function that is then callable by the user.

Question 1 - do I need to require and alias for importing a macro from a different module? If I remove the require, it complains that (despite specifying the macro explicitly with the full nested-macro path name Deribit.Helpers.Macros.generate_endpoint) it needs to require a macro. If I remove the alias, then I have to specify the entire path, which is obnoxious. Any words of advice here?

Question 2 - to get lib/deribit/helpers/macros.ex working, I needed to do some serious hackery:

defmodule Deribit.Helpers.Macros do
  alias Deribit.Helpers.Utils, as: Utils

  defmacro create_endpoint(endpoint, options \\ []) do
    quote bind_quoted: [endpoint: endpoint] do
      @endpoint Utils.scoping_workaround(endpoint)
      def unquote(Utils.format(@endpoint, [slash?: false]))() do
        get(Utils.format(@endpoint, [slash?: true]))
      end
    end
  end
end

I needed to use module variables (@endpoint) to quickly save this bound variable [endpoint: endpoint] - whats a better way of doing the above?

The idea is that in a module, I’d say “CreateEndpoint :things” and I’d have a function getthings() that I wouldn’t need to define explicitly. I don’t like the fact I’m completely hacking module variables to quickly save a variable that is clearly in scope during the bind_quoted expression, but then somehow falls out of scope once I enter the unquote block - that I then use in the string-interpolated expression… its just not good and I recognize this but it was the fastest way to get the intended behavior.

I want to learn best practices here and appreciate any feedback.

To build -

git clone https://github.com/arthurcolle/Deribit.ex deribit
cd deribit && mix do deps.get, compile
iex -S mix
~|iex|> Deribit.API.Public.getinstruments()

  1. You can utilize __using__ macro (docs are here): wrap alias in __using__ of required macro module.
    But, as mentioned in docs, it’s not best practice:

do not provide using/1 if all it does is to import, alias or require the module itself.

  1. In order to declare a module attribute, use a construction like (hope I understood your question correctly):
  ...

  defmacro __using__(_opts) do
    quote do
      Module.register_attribute(__MODULE__, :endpoint, [])
    end
  end

  ...

  defmacro create_endpoint(endpoint, options \\ []) do
    quote bind_quoted: [endpoint: endpoint] do
      @your_attribute attr_value
    end
  end

It will define an attribute @endpoint and a ‘setter’ macro function for it. Later in your macros you can get the attribute’s value just by calling @endpoint.

2 Likes

I think you might have misunderstood -

Simplest way to describe my functionality is… imagine if you wanted a function called timesX() where X can be specified dynamically by the user when they call it - so times2(5) yields 10, times5(8) yields 40. I tried to use this basic example as my go-to for when I was implemented equally simple example.
The problem is - when you’ve entered the def unquote ... end block, without specifying a module variable, it will say that “endpoint” is not defined.
Previously, instead of:

defmodule Deribit.Helpers.Macros do
  alias Deribit.Helpers.Utils, as: Utils

  defmacro create_endpoint(endpoint, options \\ []) do
    quote bind_quoted: [endpoint: endpoint] do
      @endpoint Utils.scoping_workaround(endpoint)
      def unquote(Utils.format(@endpoint, [slash?: false]))() do
        get(Utils.format(@endpoint, [slash?: true]))
      end
    end
  end
end

I had the following, which DID NOT work:

defmodule Deribit.Helpers.Macros do
  alias Deribit.Helpers.Utils, as: Utils

  defmacro create_endpoint(endpoint, options \\ []) do
    quote bind_quoted: [endpoint: endpoint] do
      def unquote(Utils.format(endpoint, [slash?: false]))() do
        get(Utils.format(endpoint, [slash?: true]))
      end
    end
  end
end

Its not exactly what I was doing before, but it definitely illustrates (just tried it in iTerm2) the scoping problems that cropped up here - maybe bind_quoted isn’t the answer, but from the options available, it definitely seemed like the best option for me. That being said, this all really needs to be simplified - unquoting/quoting is an element/simple solution but finagling with this stuff is quite obnoxious and I think the Macros documentation leaves a lot to be desired in terms of concrete/easy to wrap head around examples.

Happy to help improve the docs later but I still want to resolve my problems haha.

I think the difficulty you’re having is getting the endpoint atom to become a function name. It turns out bind_quoted won’t actually help you with that.

def is implemented so that if you just unquote an atom, it can build a function name out of it:

quote do: def unquote(:foo)(), do: "bar"

When you bind_quoted, the unquoting happens automatically within your quote for those variable names. However, the def no longer recognizes your variable as an atom to unquote, since it now looks like a normal function def:

quote bind_quoted: [name: :foo], do: def name(), do: "bar"

This is creating a function called name, not foo.

The solution is to avoid bind_quoted if you’re constructing function names and unquote manually. This works as you seem to desire:

defmodule Example.Macro do
  def format(foo, opts \\ []), do: "bar"
  defmacro create_endpoint(endpoint, options \\ []) do
    path = format(endpoint, [slash?: false])
    quote do
      def unquote(endpoint)() do
        get(unquote(path))
      end
    end
  end
end

defmodule Example do
  import Example.Macro
  create_endpoint :foo
  def get(path) do
    path |> inspect |> IO.puts
  end
end

Example.foo #=> "bar"

I do this to generate 104 Slack API endpoints more or less exactly as you are (albeit in a little more complicated fashion) in my (non-operational, still in progress) Slackkit project.

5 Likes

Use require Foo.Bar.Baz, as: Baz. Which will require the module, which is necessary, and then alias it. Alias does not require modules. You can alias to completely non existent things, such as alias Foo.Bar.Nothing, while require will explicitly check for its existence.

You need to understand the scope of the endpoint variable. Because they are both named endpoint, it does not mean they are the same. If you instead had:

endpoint = Utils.scoping_workaround(endpoint)

When you call unquote(... endpoint ...), you would be accessing the variable defined before the function. However, if you just call get(Utils.format(endpoint, [slash?: true])) inside the function, then it is going to expect an endpoint variable to be defined inside the function. Functions in Elixir start a brand new variable scope, so Elixir variables won’t automatically fallback and lookup on the module scope. That would be quite confusing as there would be a chance of accidentally calling something defined anywhere in the module. For such, Elixir has explicit mechanisms, such as the @endpoint attribute.

So you have two options: continue using @endpoint. Or also unquote(endpoint) inside the function body:

get(Utils.format(unquote(endpoint), [slash?: true]))

In fact, I’d recommend you to unquote the endpoint inside the module since all of this work could be done at compile time. Here is how I would write that code:

  defmacro create_endpoint(endpoint, options \\ []) do
    quote bind_quoted: [endpoint: endpoint] do
      noslash = Utils.format(@endpoint, slash?: false)
      slash = Utils.format(@endpoint, slash?: true)
      def unquote(noslash)() do
        get(unquote(slash))
      end
    end
  end

You may need to slightly adjust things but that would be my preferred solution.

6 Likes