mayel

mayel

Needle - Universal foreign keys, shared data fields, virtual and configurable schemas for Ecto

When using a relational DB (such an Ecto schema with a Postgres table), usually a foreign key field has to be pre-defined with a reference pointing to a specific field in a specific table.

A simple example would be a blogging app, which might have a post table with author field that references the user table.

A social network, by contrast, usually requires a graph of objects, meaning objects need to be able to refer to other objects by their ID without knowing their type.

A simple example would be likes, you might have a likes table with liked_post_id field that references the post table. But what if you don’t just have posts that can be liked, but also videos, images, polls, etc, each with their own table, but probably do not want to have to add liked_video_id, liked_image_id, ad infinitum?

We needed the flexibility to have a foreign key that can reference any referenceable object. We call our system Needle.

Note: This library was developed as part of the Bonfire project, which has some open bounties for any help improving performance:.

Besides regular schemas with universal foreign keys (Pointable), Needle provides Virtual for schemas that don’t have any fields on their own, and Mixin for storing common fields that can be re-used by multiple Pointables or Virtuals (eg. if you have multiple types that all have a name/description/body, they can share those and optionally the associated changesets and other logic).

Needle also comes with two other homegrown libraries: needle_ulid which provides a ULID datatype for Ecto (using ex_ulid) and related helpers, and exto which enables extending Ecto schema definitions in config (especially useful for adding Mixin associations to different schemas)…

https://github.com/bonfire-networks/needle

Most Liked

garrison

garrison

Your docs are very thorough and well-written :slight_smile: If I had to offer one suggestion there, I think some query examples would be helpful near the end - perhaps some real-world examples from Bonfire? But really, overall excellent docs.

Also, is there a reason you chose to go all in on ULID instead of UUIDv7, which is now standard? I assume you guys have probably been at this since before UUIDv7 was standardized, so perhaps it was too late to change?

I mention it because it probably wouldn’t be very hard to switch over since they’re both 128 bit binaries in the DB at the end of the day and have a pretty similar structure (timestamp + random). Generating a UUIDv7 in Elixir is like three lines of code, and then you can just delegate all the Ecto.Type functions over to Ecto.UUID since they’re compatible anyway.

For anyone coming across this post who’s curious, the “correct” way to do this in an RDBMS is to create a parent table (e.g. likeable) and then give each “child” table a foreign key to the parent table (e.g. likeable_id on posts). Then a user can have a like which joins through likeable to the post. This way the new foreign key columns always end up on the new table.

Indeed, if I understand correctly the above is actually exactly what this project is doing, except with lots of tooling to generalize it to the entire schema, which is cool!

mayel

mayel

With Needle, we don’t need a separate Likeable schema because all Pointables or Virtuals can be likeable by default. Let’s revise these examples to reflect how Needle can be used.

Note: this is a slightly edited version of the result of feeding Claude.AI with the above sample migrations and queries and the Needle and Exto readmes. It looks correct at a glance but please consider it pseudocode and read the docs if you want to implement something like this.

Here’s those same schemas as you might define them with Needle:

defmodule YourApp.Post do
  use Needle.Pointable,
    otp_app: :your_app,
    table_id: "P0STSC0NTENTF0RV1EW1NG000",
    source: "posts"

  pointable_schema do
    field :title, :string
    field :summary, :string
    field :content, :string
  end
end

defmodule YourApp.Video do
  use Needle.Pointable,
    otp_app: :your_app,
    table_id: "V1DE0C0NTENTF0RV1EW1NG000",
    source: "videos"

  pointable_schema do
    field :duration, :integer
  end
end

defmodule YourApp.Image do
  use Needle.Virtual, # use virtual here because schema has no fields, and uses URI mixin instead
    otp_app: :your_app,
    table_id: "1MAGEC0NTENTF0RV1EW1NG000",
    source: "images"

  virtual_schema do
  end
end

defmodule YourApp.Poll do
  use Needle.Pointable,
    otp_app: :your_app,
    table_id: "P011C0NTENTF0RV1EW1NG000",
    source: "polls"

  pointable_schema do
    field :question, :string
  end
end

defmodule YourApp.User do
  use Needle.Pointable,
    otp_app: :your_app,
    table_id: "VSERSYS1NGTHESYSTEM00000",
    source: "users"

  pointable_schema do
    field :username, :string
    field :email, :string
  end
end

defmodule YourApp.Like do
  use Needle.Pointable,
    otp_app: :your_app,
    table_id: "11KES0NBJECTSBYVSERS0000",
    source: "likes"

  pointable_schema do
    belongs_to :user, YourApp.User
    belongs_to :liked, Needle.Pointer
  end
end

# Example of a mixin that can be used across different types of content
defmodule YourApp.URI do
  use Needle.Mixin,
    otp_app: :your_app,
    source: "uri_mixin"

  mixin_schema do
    field :uri, :string
  end
end
  • We’ve removed the separate Likeable schema.
  • All Pointable or Virtual schemas (Post, Video, Image, Poll) can now be liked without needing a specific Likeable association.
  • The Like schema now has a liked field that references Needle.Pointer, allowing it to like any Pointable object.
  • We’ve added a URI mixin to show how mixins can be used to add shared fields across different types of content.

Now, let’s update our query examples to show how to use these schemas with Needle:

  1. Creating a new post:
defmodule YourApp.Posts do
  alias YourApp.{Repo, Post}

  def create_post(content, title, summary) do
      %Post{
      	content: content,
        title: title,
        summary: summary}
      |> Repo.insert()
  end
end
  1. Creating a like:
defmodule YourApp.Likes do
  alias YourApp.{Repo, Like}

  def record_like(user_id, liked_id) do
    %Like{
      user_id: user_id,
      liked_id: liked_id
    }
    |> Repo.insert()
  end
end
  1. Querying likes for a specific object:
defmodule YourApp.Likes do
  import Ecto.Query
  alias YourApp.{Repo, Like}

  def list_object_likes(object_id) do
    from(l in Like,
      where: l.liked_id == ^object_id
    )
    |> preload([:user])
    |> Repo.all()
  end
end

Next we can also add useful associations in config using Exto. Add the following to your config.exs:

config :your_app, YourApp.Post,
  has_many: [
    likes: {YourApp.Like, foreign_key: :liked_id}
  ]

Listing likes for a post using the association:

defmodule YourApp.Posts do
  import Ecto.Query
  alias YourApp.{Repo, Post}

  def list_post_likes(%Post{} = post) do
    post
    |> Repo.preload(likes: [:user])
  end

  def list_post_likes(post_id) when is_binary(post_id) do
    Post
    |> Repo.get(post_id)
    |> list_post_likes()
  end
end

Where Next?

Popular in Announcing Top

tmbb
I’ve published the first version of my Makeup library. It’s a syntax highlighter for Elixir in the spirit of Pygments, Currently it highl...
New
mischov
import Meeseeks.CSS html = HTTPoison.get!("https://news.ycombinator.com/").body for story <- Meeseeks.all(html, css("tr.athing")) do...
New
pkrawat1
Presenting Aviacommerce, open source e-commerce platform in Elixir Aviacommerce is an open source e-commerce platform in Elixir. We at...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
deadtrickster
I’ve just released stable versions of my Prometheus Elixir libs: Elixir client [docs]; Ecto collector [docs]; Plugs instrumenter/Export...
New
mikehostetler
I’m excited to announce Jido, a framework providing foundational primitives for building autonomous agent systems in Elixir. While develo...
New
zorbash
I created Kitto a framework for dashboards inspired by Dashing. The distributed characteristics of Elixir and the low memory footprint...
New
MRdotB
I needed to reuse React components from my Chrome extension in my Phoenix/LiveView backend. I noticed that for Svelte/Vue, there are live...
New
wojtekmach
Hey everyone! Req is an HTTP client for Elixir that I’ve been working on for quite some time. There is already a lot of HTTP clients out...
New
cjen07
parameterized pipe in elixir: |n> edit: negative index in |n> and mixed usage with |> are supported example: use ParamPipe ...
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

We're in Beta

About us Mission Statement