Announcing EctoFoundationDB 0.1.0!

EctoFoundationDB is an Ecto adapter for FoundationDB, a distributed key-value store that is designed to be scalable, fault-tolerant, and performant.

Quick Links:

• Documentation
• Livebook Guide
• GitHub
• Hex.pm
• Why FoundationDB

Features:

• CRUD plus indexes
• Multi-tenancy
• Automatic migrations
• Custom indexes
• FDB Transactions

Due to FoundationDB’s Layer Concept, EctoFoundationDB is a more than a wrapper. It has opinionated default behavior that is intended to fit the needs of modern web applications, and it also allows you to add structure to your data beyond the table. It does both of these things with ACID transactions, to ensure your entire data model is in a consistent state no matter what.

For example, maybe you want to put all your Users in a durable queue to process later. Or maybe you’re interested in implementing your own vector similarity search directly on top of your existing data model. Perhaps you’re intrigued by the sound of automatic schema migrations. Maybe you just need some very solid simple data storage with high availability.

EctoFoundationDB can help you do any of this.

For me, after managing various medium-to-large-scale SQL and NoSQL databases in production for 12 years and eventually deciding I’m more of a NoSQL guy, I simply wanted an Ecto adapter where I felt like I was at home.

Finally, thanks to @Schultzer and @warmwaffles for their open source adapters. I learned a lot from ecto_qlc and ecto_sqlite3, and you should definitely check them out!

Hi everyone,

EctoFoundationDB v0.6.0 is published. The focus of this release is a new module called EctoFoundationDB.Sync that represents a milestone in the read-path Sync Engine that I’ve been documenting in Livebooks over the past 9 months or so.

Inspired by what Phoenix.Sync has done for Postgres users, EctoFoundationDB.Sync offers a similar batteries-included syncing experience for those of us that wish to use FoundationDB for their apps ( => me!). The guiding principle is to declare the queries upfront and have the assigns automatically update, without PubSub.

Here’s an example LiveView showing how we can manage various syncing operations on the database as the user navigates on the page (handle_params). The “magic” auto-updating is done via careful FDB watches and LiveView’s attach_hook.

defmodule DemoLive do
  use Phoenix.LiveView

  alias EctoFoundationDB.Sync
  import Ecto.Query

  @query_catalog from(p in Product, order_by: p.name)
  @query_reviews from(r in Review, order_by: {:desc, r.inserted_at})

  def mount(_params, _session, socket) do
    tenant = Tenant.open!(Repo, "sync-sample")

    # :catalog drives our navigation bar, allowing the user to select a Product
    {:ok, socket
      |> put_private(:tenant, tenant)
      |> Sync.sync_all(Repo, :catalog, @query_catalog)}
  end

  def handle_params(%{"id" => id}, _uri, socket) do
    # When the user selects a Product, it's loaded in :product and its Reviews in :reviews
    {:noreply,
      socket
      |> Sync.sync_one(Repo, :product, Product, id)
      |> Sync.sync_all_by(Repo, :reviews, @query_reviews, product_id: id)}
  end

  def render(assigns) do
    # ...
  end

  # That's it! Really -- nothing more
end

There is yet another Livebook that demonstrates the capabilities end-to-end: Sync Engine III - Batteries Included. (Livebook is awesome btw!)

The Livebook includes the LiveView shown above as well as one that is more sophisticated using LiveComponents.

I’ve really enjoyed the database / data management discussion on the forum lately. And especially the new projects that you all continue to contribute to the community. I do a lot of lurking, and not a lot of responding, so I wanted to thank all of you for sharing your thoughts and your code.

Hi there, thanks for the message. You’ve rightly identified that there are some risks to running erlfdb or ecto_foundationdb for a project. Let’s discuss from the bottom-up.

FoundationDB server and libfdb_c: Maintained and released by the FoundationDB team at Apple. Production ready, battle tested. There are reasons to choose FDB over other DBs and reasons not to. Happy to discuss more, but probably out of scope for this post.

erlfdb: A NIF wrapper of libfdb_c. With any NIF there is risk of bringing the BEAM VM down. The project was originally implemented by the CouchDB team working closely with the FDB team. The apache-couchdb/erlfdb project is used in production apps with success.

The foundationdb-beam/erlfdb fork (where the hex.pm package comes from) has some changes, and to my knowledge has not yet had a production deployment anywhere. However, I’m aware of one project where it will be soon, in an app that’s very important to me professionally.

I’ve been conservative with my changes to the fork to preserve its production-readiness. I’m confident erlfdb will hold up well to production scrutiny. Of course please report any bugs to the issues page.

ecto_foundationdb: Still young, and ready for experimentation. No battle testing to my knowledge, but I do seek to change that. There are some projects that I have in mind, but they’re still a ways out.

In FDB parlance, ecto_foundationdb is a Layer. A consequence of an FDB stack is that correctness in the Layer is just as important as correctness in the database itself, so extensive testing is encouraged. An example of something that needs more testing focus is migrations. Everything works on paper, but it needs a longer term app to live in to make sure the migrations hold up as expected across iterative application releases.

In short, I’d call erlfdb production-ready, but not yet production-proven (due to the fork) and ecto_foundationdb is ready for community experimentation. I am personally and professionally invested in them both, and welcome further discussion, issues, and PRs.