A comprehensive approach to data loading and permissions

[axelson]

I am working on an application of mid to high size and complexity, it is an umbrella application with at least 3 main applications (but you could think of them as Phoenix Contexts if you wish). There is an existing client-facing JSON API, a little pure HTML rendering, some websockets, and an internal API (for another server to interface with us). Currently data-loading and permissions checking happens in an ad-hoc manner. As we are just starting to add GraphQL support with Absinthe I’ve been thinking of using Dataloader throughout the various applications and contexts. Does anyone have familiarity with this type of approach?

I have been fairly deeply impacted by the philosophy espoused by Dan Schafer in this GraphQL talk: https://www.youtube.com/watch?v=etax3aEe2dA

So my goals are to have entities always loaded with respect to the current viewer (usually a user) and for that data loading to live in the context of the specific application.

The main approach that has come to mind so far is to create a %UserRequest{} struct with a %Viewer{} struct embedded. The %Viewer{} would have typically have a normal %User{} associated with it but sometimes the user would be a :super_admin (i.e. me) which would have extra permissions and it might even be :other_internal_system. The %UserRequest{} struct would additionally contain the %Dataloader{} that could be used to fetch entities that the user is allowed to see.

So my context/application data fetching functions would look something like this:

defmodule Blog do
  @spec fetch_post(binary, %UserRequest{}) :: {:ok, {%Post{}, %UserRequest{}}} | {:error, term}
  def fetch_post(id, %UserRequest{loader: loader, viewer: viewer}) do
    ...
  end

  @spec fetch_posts(list(binary), %UserRequest{}) :: {:ok, {list(%Post{}), %UserRequest{}}} | {:error, term}
  def fetch_posts(ids, %UserRequest{loader: loader, viewer: viewer}) do
    ...
  end
end

So the context/application data fetching functions would take in the data to query, along with the %UserRequest{} (which now that I think of it should perhaps be called a %UserRequestContext{} or maybe just %RequestContext{}). And along with fetching and returning the actual data, they would potentially load new data into the %UserRequest{} data structure which is the reason that the %UserRequest{} needs to be returned from the data fetching method as well.

This would allow all of the data fetching methods to have a common format for permission checking and avoid loading the same data multiple times (note that the permission checking will sometimes have to load data specific to the user).

Does this approach make sense? Are there better alternatives that wouldn’t require re-writing so much of the data-fetching logic? Should I instead move the dataloader to the Process dictionary (which is effectively the approach taken by the javascript dataloader)? @jfrolich I’m especially interested in your thoughts since I think your talk (which I’m about to watch) and PR are related to this topic.

[jfrolich]

Definitely makes sense to me, in my application the contexts indeed take a user as the last argument for authorization.

It’s not really clear to me how including %UserRequestContext{} as the result of the function solves the dataloading/n+1 problem.

That is indeed exactly the problem I’m trying to solve with the PR. Interested what you think about the talk and the approach taken in the PR. Unfortunately I didn’t have much time to work on this. But help is welcome, I think it’s a pretty important element for more complex GraphQL apps.

PS: using the PR you don’t need to pass the loader, so you can just use the viewer as the last argument. Only in the graphql the deferrable is resolved using the dataloader. The deferrables are as much as possible resolved at the same time, so batching, caching and parallelization is applied.

[axelson]

@jfrolich

It’s not really clear to me how including %UserRequestContext{} as the result of the function solves the dataloading/n+1 problem.

It doesn’t really solve all of it. What it solves is the avoidance of re-fetching already fetched data (e.g. when checking permissions across multiple fields). What it gives up is effecient batching because the first time you call a method like Blog.fetch_posts/2 it would call Dataloader.run. Basically I’m trying to work out a concept for fetching data using dataloader’s existing semantics and what I’m coming to realize is that it falls short in a few key areas.

@jfrolich

That is indeed exactly the problem I’m trying to solve with the PR. Interested what you think about the talk and the approach taken in the PR. Unfortunately I didn’t have much time to work on this. But help is welcome, I think it’s a pretty important element for more complex GraphQL apps.

I do think that the approach is very interesting and has the potential to get us to where we want to go but I feel like that the semantics are awkward and feel at odds with the nature of Elixir. Of course part of that is a knee-jerk reaction because I have not actually tried using your PR.

@jfrolich

PS: using the PR you don’t need to pass the loader, so you can just use the viewer as the last argument. Only in the graphql the deferrable is resolved using the dataloader. The deferrables are as much as possible resolved at the same time, so batching, caching and parallelization is applied.

That is interesting and does in fact make the interface much easier to use (passing something like a %UserRequestContext{}) all over the place could definitely feel like needless boilerplate (and possibly error prone). But isn’t the relationship between the deferrable and the dataloader a hidden coupling? How do you ensure that the deferrable is run against the correct dataloader? Or against a dataloader with the correct source?

What if instead of introducing the concept of Deferrable we used a process? Imagine a DataloaderServer (that might be a GenServer). It would receive messages that ask it to fetch data and also messages that would ask it to return data that it has already fetched. So your context/application module might have a function that looked like this:

defmodule Blog do
  @spec load_posts(list(id), %Viewer{}, dataloader_server_pid) :: (() -> {:ok, %Post{}} | {:error, term})
  def load_posts(ids, _viewer, dataloader_server_pid) do
    DataloaderServer.load(dataloader_server_pid, {Blog, :post, id})
    fn ->
      DataloaderServer.get(dataloader_server_pid, {Blog, :post, id})
    end
  end
end

As long as not all of the code is immediately calling the 0-arity functions that it receives than the DataloaderServer can do appropriate batching (although it may need to use a mechanism like selective receive so it can batch together all of the load’s before responding to any get’s).

While this does have some similarities to your Deferrable concept I do think that the API is more inline with typical Elixir semantics.

[jfrolich]

Ah I see that it can help with caching. Batching is the most important thing to do because that is will solve the N+1 problem. Caching is just a small optimization.

Regarding solving it with processes. That is not easily possible because Elixir is lacking an event loop (which is a plus, but makes these things harder). Anyway there might be an implementation possible that solves the issue in a way I didn’t think of, interested to hear about other approaches.

You can also use Deferrables without the syntax sugar (calling Defer.then(fn ... end)), but then you have to deal with callbacks. Might be more explicit and the Elixir way.

I do think deferrables are a pretty good compromise, lazy data structures are common in functional languages and very powerful. You can compare it to the Stream as another example of a lazy data structure in Elixir.

[dimitarvp]

I have read the comments here and Dataloader’s README but I am still not quite sure what is @axelson trying to achieve… can you please elaborate? You immediately started talking about request contexts and I lost the main goal.

[jfrolich]

@axelson

But isn’t the relationship between the deferrable and the dataloader a hidden coupling? How do you ensure that the deferrable is run against the correct dataloader? Or against a dataloader with the correct source?

You need to have the right sources available when you run the deferrable indeed. But this is the same situation right now if you put the dataloader in the graphql context (need the right sources).

[jfrolich]

@dimitarvp

I have read the comments here and Dataloader’s README but I am still not quite sure what is @axelson trying to achieve… can you please elaborate? You immediately started talking about request contexts and I lost the main goal.

I did a meetup talk about this issue, I think this gives an idea about the high level problem.

Also see the more general talk of Dan Schafer.

In short Absinthe GraphQL wrote a Dataloader module that solves the n+1 problem in loading data. However their approach comes short that you now have to load data in the resolver module instead of your business logic modules.

[dimitarvp]

Apologies, I haven’t listened to the talk yet, but can you tell me, from the outset, what does your code bring in as an added value compared to doing Ecto.Multi operations inside the GraphQL endpoints?

I will check out the talk but cannot do it right now.

[jfrolich]

@dimitarvp

Apologies, I haven’t listened to the talk yet, but can you tell me, from the outset, what does your code bring in as an added value compared to doing Ecto.Multi operations inside the GraphQL endpoints?

Solve the N+1 problem. In short if you query a list of users and for each user you would like to query their best friend. You now have N+1 queries that could have been two queries if you would batch the “best friend” queries together.

[dimitarvp]

I still don’t understand. In cases like these I have something like this:

defmodule User do
  def with_best_friend(list_of_user_ids) do
    # an elaborate Ecto query here
  end
end

Then your GraphQL endpoint simply calls your convenience function that specifically takes care to avoid the N+1 query problem. Am I missing something obvious?

[jfrolich]

@dimitarvp

I still don’t understand. In cases like these I have something like this:

defmodule User do
  def with_best_friend(list_of_user_ids) do
    # an elaborate Ecto query here
  end
end

Then your GraphQL endpoint simply calls your convenience function that specifically takes care to avoid the N+1 query problem. Am I missing something obvious?

The dynamic nature of GraphQL makes it hard to hand optimize all N+1 situations.

It’s pretty easily fixed with a batch function in the resolver, or the current Dataloader. So it’s a fixed problem. The only issue is that it would be nicer to have data loading happen at the busines logic layer.

[dimitarvp]

Your goal is to have a generic solution that does not require hand-crafted queries for every endpoint, then?

[jfrolich]

@dimitarvp

Your goal is to have a generic solution that does not require hand-crafted queries for every endpoint, then?

Yeah the general idea is to have a nice API where you can naively call the business logic module, where the N+1 problem in data fetching is automatically optimized away. Such as:

users = Enum.map(user_ids, &User.get(&1))

The PR explores a Deferrable lazy data structure that only resolves data fetching once you run it.

[dimitarvp]

Maybe I should stop asking questions and go study the PR.

Sounds a bit like magic. I like your idea.

That being said, I get a PTSD attack when I hear about request contexts… Java flashbacks.