A design pattern for more flexible context APIs

A question I had when first learning contexts and Ecto was how to expand the default context API to support more flexible queries. Usually your auto-generated context modules look like:

defmodule App.Accounts do
  def list_users()
  def get_user(id)
  def get_user!(id)

  ...
end

Which soon becomes a little restrictive when you need to query your database with different filters. How do I find a user by email? How do I list only admin users? So one would create helper functions such as:

defmodule App.Accounts do
  def list_users()
  def list_admin_users()
  def get_user(id)
  def get_user!(id)
  def get_user_by_email(email)
  def get_user_by_email!(email)

  ...
end

As you can imagine, creating a function for each possible query type doesn’t scale very well, and if you introduce GraphQL to your application, where queries tend to be both user-generated and composable (find a user by email which is an admin), this way of organising your context API gets problematic.

Thankfully, the dataloader library, which is often used in conjuction with (Absinthe) GraphQL, shows an interesting pattern that can be used in all cases, not only in the GraphQL context.

dataloader mandates the creation of a query/2 function, that takes a queryable (i.e. an Ecto schema or a query) and a list of filters we want to apply to that queryable. Basically we want to take a list of filters, and build a query out of it.

def query(User, opts \\ [])
  # apply filters listed in opts and return an Ecto query
end

We can use this function to make all our context functions accept a list of filters than can be composed together:

defmodule App.Accounts do
  def query(User = queryable, opts \\ []) do
    Enum.reduce(queryable, opts, fn 
      {:email, email}, query -> 
        where(query, [u], u.email == ^email)

      {:admin?, admin?}, query -> 
        where(query, [u], u.admin? == ^admin?)

      {:in_group, group_name}, query -> 
        query
        |> join(:inner, [u], g in assoc(u, :group))
        |> where(g.name == ^group_name)
    end)
  end)

  def list_users(opts \\ []) do
    User
    |> query(opts)
    |> Repo.all()
  end

  def get_user(opts \\ []) do
    User
    |> query(opts)
    |> Repo.one()

  ...
end

All the possible filters are now encapsulated in this query function, they’re composable because Ecto is awesome, and our context API remains slim and neat. Want to find an admin user by email? Accounts.get_user(email: "foo@example.com", admin?: true). And bonus point, your context module is ready to be integrated with your GraphQL server.

An astute reader might notice that some of the filters tend to be very similar: we often want to filter a field by value (i.e. field == something), we often want to give our context API a way of ordering the results or limiting the number of rows returned. It would be nice to generalise some of these filters in a shared module. What I tend to do in my code is change the query function like this:

  def query(User = queryable, opts \\ []) do
    Enum.reduce(queryable, opts, fn 
      {:in_group, group_name}, query -> 
        query
        |> join(:inner, [u], g in assoc(u, :group))
        |> where(g.name == ^group_name)

      filter, query -> Helpers.default_query_filters(filter, query)
    end)
  end)

# in some other helper module...

def default_query_filters({:preload, preloads}, query) do
  preload(query, ^preloads)
end

def default_query_filters({:limit, n}, query) do
  limit(query, [q], ^n)
end

def default_query_filters({field, value}, query) when is_atom(field) do
  where(query, [q], field(q, ^field) == ^value)
end

The query function in the context now only contains filters and queries specific to the module, whereas the common functionality such as preloading or filtering a field by value are held by a helper module, since the logic is always the same no matter the queryable. And all this work lets us express complex queries very simply: Accounts.list_users(in_group: "Friends", admin?: true, preload: :group)

I’ve never seen this pattern in the wild, and I’ve been using it for a year at work with great success. I hope this helps somebody, and if someone has more free time than me, I imagine the default_query_filters function could be implemented in a library available for everybody to use.

19 Likes

This is how I’ve been programming my big project to access an old DB with lots and lots of joins to do even trivial lookups. ^.^;

I haven’t really generalized any of the lookups as if it’s simple like that I’ll just do it on-site instead, the actual ‘query’ function holds anything that’s not generic.

2 Likes

I use a similar pattern where I base my queries on this ListQuery implementation (https://github.com/SalesLoft/okr-app/blob/master/lib/okr_app/query/list_query.ex)

This lets me turn a map of filters into an Ecto.Query really easily, which I often use for something like def all(params). I will often write context functions that are more specific and produce the map for you, like all_for_user may take a User and create the right params for the query.

There is a downside to this approach, which is that it can be difficult to answer the question “What queries, exactly, will my system produce?” This question is important when considering indices, or if deprecating some function.

2 Likes

This is similar to what my token_operator (GitHub - baldwindavid/token_operator: Helper to make clean keyword APIs to Phoenix context functions) package does though a bit more opinionated and probably easier to understand.

It is for this reason that I’ve mostly moved away from using the pattern in the controller in favor of explicit function references.

# controller
Calendar.list_reservations([
  &Calendar.filter_reservations_by_room(&1, rooms),
  &Calendar.filter_reservations_by_day(&1, datetime),
  &Calendar.preload_reservation_owner_and_company/1,
  &Calendar.preload_reservation_room_and_location/1
])

In the context, I just run each of those functions in succession.

# context
def list_reservations(queries \\ []) do
  Reservation
  |> (fn query -> Enum.reduce(queries, query, & &1.(&2)) end).()
  |> Repo.all()
end

Okay, really I wrap that ugly reduce part in a utility, but you get the idea.

5 Likes

Do you have any scenarios that take user input and provide queries against it? The best example I have to demonstrate this would be a /feed controller that can take a variety of filters (type, occurred_at, resource_type, etc). Would you convert the user’s params into the functions via something like pattern matching?

1 Like

I haven’t actually had the need for that, but assume will at some point. If you’re talking about doing it with the pattern using explicit function references, perhaps something like:

# controller
Syndication.list_feeds([
  &Syndication.filter_and_sort_feeds_by(&1, params)
])
# context
def filter_and_sort_feeds_by(query, attrs \\ %{}) do
  attrs
  |> Map.take(["type", "occurred_at", "resource_type", "sort_by"])
  |> Enum.map(fn {key, value} ->
    case key do
      "type" -> &filter_feeds_by_type(&1, value)
      "occurred_at" -> &filter_feeds_by_occurred_at(&1, value)
      "resource_type" -> &filter_feeds_by_resource_type(&1, value)
      "sort_by" -> &order_feeds_by(&1, value)
    end
  end)
  |> Enum.reduce(query, & &1.(&2))
end

That provides the ability to use it just like the other filters. For something that custom though, maybe simpler to just have a dedicated function that actually hits the repo.

# controller
Syndication.filter_and_sort_feeds(params)
# context
def filter_and_sort_feeds(attrs \\ %{}) do
  attrs
  |> Map.take(["type", "occurred_at", "resource_type", "sort_by"])
  |> Enum.map(fn {key, value} ->
    case key do
      "type" -> &filter_feeds_by_type(&1, value)
      "occurred_at" -> &filter_feeds_by_occurred_at(&1, value)
      "resource_type" -> &filter_feeds_by_resource_type(&1, value)
      "sort_by" -> &order_feeds_by(&1, value)
    end
  end)
  |> Enum.reduce(Feed, & &1.(&2))
  |> Repo.all()
end
5 Likes

Can someone explain to me what this pattern matching def query(User = queryable in function declaration does and why is it used there?

1 Like

It makes sure to accept only the User module as first argument, and all Ecto.Schema modules are queryable.

Why does it pattern match on that schema? Because a context might be working with more than one queryable. The context shown in the example is called App.Accounts so it’s not unreasonable to have:

def query(User = queryable, opts)
  # User own queries
end

def query(Group = queryable, opts)
  # Group own queries
end

def list_users(opts \\ []), do: Repo.all(query(User, opts))
def list_groups(opts \\ []), do: Repo.all(query(Group, opts))
2 Likes

Oh true because module names are atoms, sweet!

I toke this great example and found 2 issues?

  • Shouldn’t queryable & opts be exchanged in Enum.reduce/3?
  • The end of query/2 shouldn’t have a )?
defmodule App.Accounts do
  def query(User = queryable, opts \\ []) do
    Enum.reduce(opts, queryable, fn
      {:email, email}, query ->
        where(query, [u], u.email == ^email)

      {:admin?, admin?}, query ->
        where(query, [u], u.admin? == ^admin?)

      {:in_group, group_name}, query ->
        query
        |> join(:inner, [u], g in assoc(u, :group))
        |> where(g.name == ^group_name)
    end)
  end
end

But thanks a lot for posting this. Very useful for me!

1 Like

They indeed should! Thanks for the corrections, I’ll correct my post.

EDIT: whoops, I’m unable to edit the post, I guess those are left as an exercise for the reader :slight_smile:

This pattern is very similar to ActiveRecord, the ORM from Ruby on Rails framework.

Basically, you can do things like this

User
  .where(is_admin: true)
  .order(email: :asc)
  .update_all(ia_admin: false)

The context approach is something different, you abstract all the query construction behind the semantic function name and you can explore all the queries just looking into context files.

If you’re going to create “meta” contexts then we’ll go back to the ActiveRecord pattern where queries are constructed elsewhere in your codebase but not in one single place.

Both approaches have pros and cons but I don’t think mixing them is a good idea. For me it’s two completely different things because in one case you have “the right way” and you’re overindulging yourself in all king of helpers and pre-generated code.

I can’t find it right now but can remember when rails had code like def seven(), do: 7 and you could write seven.days.ago or something similar. It does create pleasant experience but empirically it’ll end up in a codebase which is implicit and challenging to maintain.

On the other side, I comprehend Elixir approach like be explicit, be flexible, allow user decide what is best and how to approach problems and it work best when you’re experienced engineer. Finally, we have configurable, explicit and flexible interfaces in Elixir ecosystem.

I haven’t used ActiveRecord in a decade, but the point of contexts is separating the business logic from the rest of the application. Multiple query functions or the design pattern described here are still miles better than the regular usage of ActiveRecord, in that it is usually called in the controllers, making tests more difficult as you then require a database connection.

This pattern comes natural if you need to integrate Absinthe GraphQL + dataloader in your application, and it made sense to me to use the same logic even for regular Ecto consumers that are not necessarily connected to Absinthe.

In any case, as with most design patterns, there is No True Way, and I find having a small amount of functions in my contexts better for tests, as contexts are usually the API/contract to test against, and the fewer the functions, the easier it is to mock against.

1 Like