How to expose (or use) a module attribute that is built using macros

Hi everyone, I was toying around with macros today and I reached a stagnation point. After trying multiple approaches I started to think that what I’m doing might not be possible, so I’m hoping for some guidance and/ or alternatives.

I’m trying to implement a module that will help me define some counter caches in a table. Here’s the general idea… I have a schema called reactions that stores various kinds of interactions a user might provide for a post:

schema "reactions" do
    field :feeling, Ecto.Enum, values: [:like, :dislike]
    # embeds_one :counter_caches, Cache
end
post_id user_id feeling
1 1 like
1 2 dislike
1 3 like
2 1 like

I expect that for those enum values, the following fields would be generated in the schema: feeling_like_count and feeling_dislike_count. Here’s what I came up with:

defmodule CounterCache do
  import Ecto.Query

  defmacro __using__(_opts) do
    quote do
      import CounterCache

      Module.register_attribute(__MODULE__, :counter_cache_fields, accumulate: true)
    end
  end

  defmacro counter_cache_field(field, opts \\ []) do
    {group, opts} = Keyword.pop(opts, :group)
    {suffix, _opts} = Keyword.pop(opts, :suffix, "count")

    quote do
      name =
        "#{unquote(group)}_#{unquote(field)}_#{unquote(suffix)}"
        |> String.trim_leading("_")
        |> String.to_atom()

      Module.put_attribute(__MODULE__, :counter_cache_fields, {unquote(group), name})
      Ecto.Schema.field(name, :integer, default: 0)
    end
  end

  defmacro counter_cache_field_enum(module, field) do
    quote do
      values = Ecto.Enum.values(unquote(module), unquote(field))
      Enum.each(values, &counter_cache_field(&1, group: unquote(field)))
    end
  end
end

So, the part that I’m stuck at is generating and exposing the query that retrieves the counter cache fields. I expect the query to be something like this:

select
  count(1) filter (where feeling = 'like') as likes
  count(1) filter (where feeling = 'dislike') as dislikes
from reactions
post_id likes dislikes
1 2 1
2 1 0

Here’s what I’ve managed to do so far with this ancillary function:

def __query__(module, fields) do
  quote bind_quoted: [module: module, fields: fields] do
    Enum.reduce(fields, from(module), fn 
      {nil, field}, query -> 
        select(query, [m], filter(count(1), not is_nil(field(m, ^field))))
      {field, value}, query ->  
        select(query, [m], filter(count(1), not is_nil(field(m, ^field) and field(m, ^field) == ^value)))
     end)
  end
end

I was hoping to be able to call a function where I’d pass the source module (where the query will fetch the information) and receive a query that I can use later to update the embed that holds the cached values in the posts table.

Repo.all(Post.counter_cache_query(Reactions))
#=> [
#=>  %{post_id: 1, likes: 2, dislikes: 1}, 
#=>  %{post_id: 2, likes: 1, dislikes: 0}
#=> ]

I had various problems trying to implement this function. I started defining it inside the __using__ macro, and had @counter_cache_fields be empty by the time I tried to generate the query. Also, tried to call the attribute outside the definition and received an error telling me that it cannot be invoked outside of the module.

So, I’m certainly missing something here, I also remembered that Ecto does something similar: ecto/schema.ex at b69d1085cfd491a859f1be36463afcf4838e4891 · elixir-ecto/ecto · GitHub with the @changeset_fields attribute; so I’m not sure exactly what’s the problem. Is what I’m trying to achieve even possible?

1 Like

Not only outside the module, neither inside after the module has been compiled.
That is why you need a helper like Ecto does in the example defining __changeset__/0

Btw, where is the function that uses the stored values in @counter_cache_fields?

1 Like

If you register the attribute with persist: true you can get at those attributes after compilation using <Module>.__info__(:attributes)

4 Likes

thank you @ityonemo TIL,
I guess I was assuming all attributes were acceded with @, but I can see that was a wrong assumption.

1 Like

Hi @eksperimental, I’m not sure I understood what you mean, but the attribute @counter_cache_fields could be passed to __query__/2 as the fields param, for instance. I purposefully didn’t include the function because I didn’t know where exactly to put it, but if you care to leave an example I’d appreciate that.

Interesting, I read the docs but it wasn’t immediately clear what this option did tbh. I’ll test that trick but I’d like to understand first if that’s really necessary (since Ecto doesn’t use this, I figure perhaps there’s another way).

Please share a gist with your code.

The code is in the post, that’s how far I got with it… The CounterCache module is the one I intend to use inside schemas to expose the query function I mentioned (which is where we’re at).

how do you call __query__(module, fields)
If you can share the module where you call use ?

Just by reviewing your code (I haven’t run any code yet), i can tell a few things.:

  • Enum.each/2 is used for side effects where you don’t expect any value to be returned. It always return :ok. So the second clause of your counter_cache_field macro will always return :ok. You probably are looking for for and make sure it is valid the desired AST what you return.
  1. def __query__(module, fields) do is a function and you are returning a quoted expression. Are you sure about this? I don’t know how you are calling this. Usually you create these helpers and call them within the macro you are building.

  2. I fail to see how you read the stored counter_cache_fields attributes.
    That is why I am asking you for the code

2 Likes

My advice is: Create a module that stores and read the attribute and does something similar to what you want to do, but without the Ecto layer.
Once you managed to properly register attributes and access them, port that code to interact with Ecto.

1 Like

Hey @eksperimental, let me try to improve on the latest comment then…

I’m not calling it yet, I just left the example of the API I want to consume to make the use case a little clearer. It should be something like this: Post.counter_cache_query(Reactions), where counter_cache_query/1 is a function that is defined by CounterCache (perhaps) which is then use-d by a hypothetical Post module… If you care to take a look at my previous examples, you’ll see that I mentioned trying to define this function on the __using__ macro only to find that the attribute was actually empty.

What you mentioned makes complete sense, but if you run the code I think it works (and now I’m curious about it as well, I’d have to make some more tests to confirm).

Like I mentioned previously, this is just “pseudo-code”, to make the spec clearer. I was defining counter_cache_query and using this function as a helper in my tests (hence the quoted expression).

This is exactly the part I’m seeking help to accomplish :sweat_smile:. What I’ve tried before was something like this if I recall properly (tried so many different approaches I won’t remember everything right now):

defmacro __using__(_) do
  quote do
    # ...
    defmacro counter_cache_query(module) do
        # CounterCache.__query__(unquote(module), @counter_cache_fields)
    end
  end
end

I think there’s a gap in my macro knowledge that I’m failing to transmit properly, and I believe that even though this makes complete sense to you, I’m not sure what you mean by that. However, I can tell you this: If you test the code, you’ll see that registering the module attribute and defining the Ecto fields in the schema works properly (more or less), so the difficulty is actually accessing the @counter_cache_fields module attribute. A simple example to visualize it would be something like this (except the function should be injected inside the module that uses it):

defmodule Module do
  @counter_cache_fields [{:feeling, :dislike_count}, {:feeling, :like_count}]

  def counter_cache_query(module) do
    Enum.reduce(@counter_cache_fields, from(module), fn ->
      # generate query using field definitions inside attribute
    end
  end
end
1 Like

In general the pattern for a macro that creates a module attribute in the calling code that is then accessed in the macro is something like this:

defmacro my_macro(opts) do
  # Set up the module and fields from wherever they are derived
  module = Keyword.get(opts, :module)
  fields = Keyword.get(opts, :fields)

  # Register and set the module attribute in the calling module at
  # macro expansion time. At this time the calling module is
  # still being compiled
  calling_module = __CALLER__.module
  Module.register_attribute(calling_module, :counter_cache_fields)
  Module,put_attribute(calling_module, :counter_cache_fields, fields)

  # Now the code that is injected in the caller
  quote do
    defmacro counter_cache_query(module) do
      # CounterCache.__query__(unquote(module), @counter_cache_fields)
    end
  end
end
1 Like

Hey @kip, one question: If I understand it right, __MODULE__ inside a quoted expression in a macro will expand to the invoking module like __CALLER__ outside of the quoted expression. So, I’m not sure I understood the difference, but I guess there’s a compilation order, would you mind explaining it a little bit more?

1 Like

__MODULE__ always refers to the module being compiled. So when __MODULE__ is inside a quote block (assuming the macro returns the quote block), then __MODULE__ is expanded in the calling module. So yes, you are correct. In this case, __CALLER__.module in a macro outside a quote block is the same as __MODULE__ inside a quote block.

In your example, we need to access the caller outside the quote block. Because we need to guarantee that the module attribute is registered and set before the macro code is expanded in the caller. Therefore it needs to be executed in the context of the macro, not the context of the caller. And therefore we need to use __CALLER__.module approach.

Does that help? I find it useful to remember a few simple things about macros:

  • A macro is a function that accepts AST as parameters and is expected to return AST
  • A quote block returns AST and is the most common way to return AST from a macro. But its otherwise not special - you can quote in any code you like at compile time or runtime.
  • A macro is expanded at the time at which it is called. Macro expansion happens recursively until there is nothing left to expand.
  • A call to a macro replaces the macro call with the result of the macro call. It is interpolated at the calling site. And then compilation proceeds normally.
  • A macro is just normal Elixir code except for the requirement to accept AST and return AST so it can execute whatever you want - recognising that this code is executing during macro expansion at compile time, not at runtime.
3 Likes

Thanks, @kip, I think this is really helpful, but I still have some questions, for example:

defmodule Greeter do
  defmacro greeting(name) do
    quote do
      Module.put_attribute(__MODULE__, :greeting, unquote(name))
    end 
   end
   
   defmacro __using__(opts) do
    quote do
      import Greeter
      Module.register_attribute(__MODULE__, :greeting, accumulate: false)
      
      def greet_inside() do
        @greeting
      end      
    end
   end 
end
defmodule Gentlemen do
  use Greeter
  
  greeting "Hello Sir"
  
  def greet, do: @greeting  
end

So, let’s see if I understand it right, the previous example will yield:

Gentlemen.greet
#==> `"Hello Sir"`
Gentlemen.greet_inside
#==> nil

Considering that a macro is expanded when it’s called, so Greeter.__using__ is expanded first, adding the @greeting attribute to Gentlemen and defining the greet_inside() function, which outputs the @greeting attribute. So, greet_inside() outputs nil when its called because at compilation time the attribute was registered but no value was assigned to it yet. Then, the greeting macro is expanded and it adds "Hello Sir" to the @greeting attribute, and that’s why greet() outputs "Hello Sir". Is my understanding right? (does the code expands top to bottom sequentially like this?).

Considering I’m right, for this to work, greet_inside() has to be a macro otherwise @greeting would be evaluated to nil at that point. So I was expecting this to work:

defmacro greet_inside() do
  quote do
    @greeting
  end
end      
import Gentlemen
Gentlemen.greet_inside()

** (ArgumentError) cannot invoke @/1 outside module
    (elixir 1.13.1) lib/kernel.ex:6111: Kernel.assert_module_scope/3
    (elixir 1.13.1) expanding macro: Kernel.@/1
    iex:12: (file)
    expanding macro: Gentlemen.greet_inside/0
    iex:12: (file)

But it won’t, because when greet_inside() is imported and called (and expanded), I’m outside of a module (I’m testing it on iex) and I can’t access attributes outside of modules. I imagine that __CALLER__.module in that context won’t return anything either, so what are my other options?

I think I understood what you mean, but I’m having a hard time applying it for this scenario. I think it’s a little different because we are setting the attribute value after the macro that registered it? (unless macros are expanded all at once, so I wouldn’t have the reference to the attribute yet!?)

1 Like

I realise this may just be an example, but in this example I don’t see what they need to use a module when import will work just as well and is clearer in most cases. Simplifying your example we get:

defmodule Greeter do
  defmacro greeting(name) do
    caller = __CALLER__.module

    # Define the attribute and set it in the calling module
    Module.register_attribute(caller, :greeting, accumulate: false)
    Module.put_attribute(caller, :greeting, name)

    # Code interpolated into the
    # calling site
    quote do
      def greet_inside() do
        @greeting
      end
    end
  end
end

defmodule Gentlemen do
  import Greeter

  greeting "Hello Sir"

  def greet, do: @greeting
end

And in execution:

iex(1)> Gentlemen.greet_inside
"Hello Sir"
iex(2)> Gentlemen.greet       
"Hello Sir"

Hopefully this helps clarify the order of expansion, compilation and execution.

2 Likes

If you feel compelled to go the “use” route, then all you would need to do is:

defmodule Greeter do
  defmacro __using__(_opts) do
    import Greeter
  end

  defmacro greeting(name) do
    ....
  end
end
2 Likes

Last thought, you might be wondering why this didn’t work:

   defmacro __using__(opts) do
     quote do
       import Greeter
       # This code will be inserted in the calling site and it will
       # be executed at runtime, not compile time because it is
       # in the quote block that will be inserted at the calling site
       Module.register_attribute(__MODULE__, :greeting, accumulate: false)
      
       def greet_inside() do 
         # Using @module_attribute requires that the module
         # attribute exist *at compile time*. But the code above is
         # only executed at runtime. Therefore the reference to 
         # @greeting will fail
         @greeting
       end      
     end
   end 
2 Likes

Hummmm, very very interesting… Thanks a lot for the explanations @kip! I think your example without use in particular made it a lot easier for me to understand. Also, the explanation about what is evaluated at runtime vs compile-time was on point, thanks a lot. I’ll start from here and I’ll read more about macros with that in mind, thanks a lot for sparring the time to educate me on the subject.

Cheers!

If perhaps someone finds this post in the future, I’ll leave a link to a gist with my final solution:
Elixir's Ecto Counter Cache Manager · GitHub. Many thanks to @kip in special and the awesome Elixir documentation on the subject of macros.

Obs.: Working with macros is very fun, especially because Elixir makes metaprogramming very enjoyable compared to other languages. However, there’s a reason why the documentation warns us to proceed with caution haha :sweat_smile:.

1 Like