Need help adding relationship support to Sanity DataLayer

For loading related data, data layers are not made aware that is loading a relationship, but rather a filter is passed down to the relationship based on the source data. So if there are no posts, no query is made to fetch the comments. You’d get a filter like post_id in [....] to your data layer for the comments query.

Thanks Zach! That makes sense. In my test I wasn’t returning any posts in the response which is why I wasn’t seeing the subsequent request for the comments.

So, this way of handling relationships makes total sense for a relational database model where the resource on the “belongs to” side of a “has many” relationship has a foreign key pointing back at the parent record. Unfortunately Sanity stores their data in a NoSQL data lake where the parent resource keeps a reference to all the related resources so when you query for a list of posts you get something back like the following:

[
  {
    _id: "7d10669d-7e9f-4c8c-a7db-e1ea369e7055",
    _type: "post",
    title: "My First Post",
    body: "A very thought provoking post",
    comments: [
      {
        _key: "13ad3eba8563",
        _type: "reference",
        _ref: "1ccb9a37-c0f2-4aa7-82cb-3e16e6ecab99"
      },
      {
        _key: "d8c1e6a53174",
        _type: "reference",
        _ref: "fff10398-7cb1-4d9f-8dfd-42d0f8aca798"
      },
      {
        _key: "6faf4ddeef52",
        _type: "reference",
        _ref: "a7e06b12-561e-4ad8-9756-cd1979e754be"
      },
    ]
  },
  … (more posts)
]

If you want to load/dereference the comments you would do so when you query for the Posts or you’d have to look at the IDs in the comments array then do a query like id in [...] on the comment resource. I’ve been trying to figure out what the best way would be to model something like that using Ash resources and at this point I have two thoughts:

1. Create a custom ref type for the comments attribute. Something like:

attributes do
  attribute :comments, {:array, ref: Comment}
end

I’d then need to handle that attribute type when casting attributes in my DataLayer which probably isn’t ideal.

2. Create a new references block on resources that use AshSanity which could look something like:

references do
  has_many :comments, Comment
end

This approach feels like the more idiomatic way to do it but may involve touching a lot of plumbing and figuring out how to add support to Ash.Query so it knows how to load those references.

It may also just be the case that the Ash DataLayer model isn’t a good fit for modeling a NoSQL data store and I’m just totally barking up the wrong tree.

Would love to get your thoughts.

Interesting. So you could do this today with calculations or manual relationships (where your tool provides the implementatiton). However, it would be good to generalize this. The goal in Ash is to act as a superset of multiple kinds of storage layers. I think we should consider adding new ttype sof relationships. Specifically references_many, where the list of ids are stored somewhere on the source record. Thoughts?

Nice. I like the idea of adding something like references_many to handle cases like this. I wouldn’t want the implementation to be tied to the shape of the data coming back from Sanity though. We’d probably want to pass the responsibility of managing the relationship down to the DataLayer somehow. For example Sanity returns what is basically an array of reference objects which contain the ID of the referenced document under the _ref key whereas other databases may return just a list of IDs or have the ID under a different key.

I’d be more than happy to help contribute to adding support for this. In the meantime I’ll take a look at how this might look using calculations or manual relationships as a stopgap until there’s builtin support at the framework level.

Yep, I think references many would point to an attribute or calculation that is expected to contain the list of keys used to look up the destination records, so could be generalized to any data layer. Then you’d provide a “references” calculation.

I think that makes sense. I have two thoughts:

1. It would be nice if we could somehow define that calculation at the DataLayer level and then also override it in the references_many. For AshSanity, the path to the ID in the reference would be the same for all references_many relationships and it would be a bummer to have to define that on every relationship. I know there are some info callbacks that are used with the DataLayers, I wonder if that mechanism could be used to supply the default calculation/lookup function or if there is a better mechanism to do that.

2. We should also implement a corresponding references_one relationship for completeness.

Well, technically belongs_to is references_one. We could potentially make it part of the data layer callbacks to provide a default implementation of references_many, or something along those lines. But we’d also want to allow it to be overridden.