Ecto_foundationdb - An Ecto Adapter for FoundationDB

Announcing EctoFoundationDB 0.1.0!

EctoFoundationDB is an Ecto adapter for FoundationDB, a distributed key-value store that is designed to be scalable, fault-tolerant, and performant.

Quick Links:

• Documentation
• Livebook Guide
• GitHub
• Hex.pm
• Why FoundationDB

Features:

• CRUD plus indexes
• Multi-tenancy
• Automatic migrations
• Custom indexes
• FDB Transactions

Due to FoundationDB’s Layer Concept, EctoFoundationDB is a more than a wrapper. It has opinionated default behavior that is intended to fit the needs of modern web applications, and it also allows you to add structure to your data beyond the table. It does both of these things with ACID transactions, to ensure your entire data model is in a consistent state no matter what.

For example, maybe you want to put all your Users in a durable queue to process later. Or maybe you’re interested in implementing your own vector similarity search directly on top of your existing data model. Perhaps you’re intrigued by the sound of automatic schema migrations. Maybe you just need some very solid simple data storage with high availability.

EctoFoundationDB can help you do any of this.

For me, after managing various medium-to-large-scale SQL and NoSQL databases in production for 12 years and eventually deciding I’m more of a NoSQL guy, I simply wanted an Ecto adapter where I felt like I was at home.

Finally, thanks to @Schultzer and @warmwaffles for their open source adapters. I learned a lot from ecto_qlc and ecto_sqlite3, and you should definitely check them out!

Nice! I was looking at possibly building a FoundationDB adapter, but I haven’t used it before to really know what I was getting myself into.

This looks awesome.

EctoFoundationDB 0.2.0 is released (changelog)

There are 2 new features for writing fast transactions:

• Pipelining: The technique of sending multiple queries to a database at the same time, and then receiving the results at a later time, still within the transaction. In doing so, you can avoid waiting for multiple network round trips. EctoFDB provides an async/await syntax on the Repo.
• Upserts: Support for Ecto options :on_conflict and :conflict_target

For example, we can combine these features into the transaction below, which safely transfers 1 unit from Alice’s balance to Bob’s balance. With Pipelining and Upserts, there are 2 waits for data from the network (best case).

(Reminder: FoundationDB’s transactions are ACID and globally serializable)

def transfer_1_from_alice_to_bob(tenant) do
  Repo.transaction(fn ->
    a_future = Repo.async_get_by(User, name: "Alice")
    b_future = Repo.async_get_by(User, name: "Bob")

    # 1. wait (Alice and Bob pipelined)
    [alice, bob] = Repo.await([a_future, b_future])

    if alice.balance > 0 do
      # No wait here (because of `conflict_target: []`)
      Repo.insert(%User{alice | balance: alice.balance - 1}, conflict_target: [])
      Repo.insert(%User{bob | balance: bob.balance + 1}, conflict_target: [])
    else
      raise "Overdraft"
    end

  # 2. wait (transaction commit)
  end, prefix: tenant)
end

Compare with this logically equivalant transaction, implemented without pipelining nor upserts. It waits 5 times for data on the network.

def transfer_1_from_alice_to_bob_but_with_more_waiting(tenant) do
  Repo.transaction(fn ->
    # 1. wait
    alice = Repo.get_by(User, name: "Alice")

    # 2. wait
    bob = Repo.get_by(User, name: "Bob")

    if alice.balance > 0 do
      # 3. wait
      Repo.update(User.change_balance(alice, -1))

      # 4. wait
      Repo.update(User.change_balance(bob, 1))
    else
      raise "Overdraft"
    end

  # 5. wait (transaction commit)
  end, prefix: tenant)
end

Thanks for reading!

EctoFoundationDB 0.3.0 is released (changelog)

Breaking changes

We’ve refactored the implementation of multitenancy, making 0.3.0 incompatible with data from previous versions. If you have a database that needs to be upgraded, please submit an issue.

New feature: Watches

FoundationDB Watches are similar to Triggers in an RDBMS. Registering a watch on a particular key provides a guarantee* that when that key in the database is changed, the client application is notified with a push-style notification, delivered directly to the Elixir process that requested it.

future = Repo.watch(struct, label: :mystruct, prefix: tenant)
# later on...
receive do
  {ref, :ready} ->
    # `struct` changed
end

Livebook | Watches in LiveView has a demonstration of using Watches instead of PubSub with a simple phoenix_playground app.

EctoFoundationDB 0.4 is released (changelog)

New feature: Large Structs

In a FoundationDB key-value pair, a value is limited to 100,000 Bytes. Previously EctoFoundationDB did not protect your app from this limitation. Now, we will split the binary among several keys behind the scenes. No changes to the API surface. Note: other FDB limitations still apply.

Love your work!
I’ve been following the project closely. In fact, the idea of using Fdb in a real project is very enticing.
However, I’m a bit worried about bus factor of 1 and basically no community.
Can you provide your thoughts on this ? How mature is the whole solution ? I’m not asking about Fdb itself, which is battle-proven and is very stable, obviously, but specifically about using it with Elixir. If I understand correctly, the C driver is provided by the Fdb itself (Apple ?), so it must be stable. However, there’s also erlfdb.
Fdb seems like an amazing technology, pity that more companies don’t want to invest in it.

Hi there, thanks for the message. You’ve rightly identified that there are some risks to running erlfdb or ecto_foundationdb for a project. Let’s discuss from the bottom-up.

FoundationDB server and libfdb_c: Maintained and released by the FoundationDB team at Apple. Production ready, battle tested. There are reasons to choose FDB over other DBs and reasons not to. Happy to discuss more, but probably out of scope for this post.

erlfdb: A NIF wrapper of libfdb_c. With any NIF there is risk of bringing the BEAM VM down. The project was originally implemented by the CouchDB team working closely with the FDB team. The apache-couchdb/erlfdb project is used in production apps with success.

The foundationdb-beam/erlfdb fork (where the hex.pm package comes from) has some changes, and to my knowledge has not yet had a production deployment anywhere. However, I’m aware of one project where it will be soon, in an app that’s very important to me professionally.

I’ve been conservative with my changes to the fork to preserve its production-readiness. I’m confident erlfdb will hold up well to production scrutiny. Of course please report any bugs to the issues page.

ecto_foundationdb: Still young, and ready for experimentation. No battle testing to my knowledge, but I do seek to change that. There are some projects that I have in mind, but they’re still a ways out.

In FDB parlance, ecto_foundationdb is a Layer. A consequence of an FDB stack is that correctness in the Layer is just as important as correctness in the database itself, so extensive testing is encouraged. An example of something that needs more testing focus is migrations. Everything works on paper, but it needs a longer term app to live in to make sure the migrations hold up as expected across iterative application releases.

In short, I’d call erlfdb production-ready, but not yet production-proven (due to the fork) and ecto_foundationdb is ready for community experimentation. I am personally and professionally invested in them both, and welcome further discussion, issues, and PRs.

This is a really cool project, and I think most people passing by this thread don’t realize how much work this actually is. You’re essentially building a database inside an Ecto adapter

If I could offer a couple points of feedback:

First, in your docs (which are great btw) you mention that users should either use a secondary index to perform :where queries or instead implement the filter themselves with Enum.filter (or Stream.filter).

I think this is a mistake. The problem is that if someone writes code to perform a :where query and then filter it, and then they decide they want to improve performance with an index, they have to rewrite and retest their code. If you instead implement the (local) filtering inside the adapter, they can keep their code the same and add indexes as needed. This is more work, of course, but I think it would be worth it, because I could see this situation coming up a lot.

Second, I noticed you’re storing records in the DB via term_to_binary on the structs (from what I can tell it’s actually kw lists but I didn’t look too deep).

The problem is that if you encode the k/v pairs of the records with the actual keys, they can never be updated. You will never be able to rename or delete columns without rewriting the entire table, which is not viable at scale because the table will be too big to rewrite atomically.

Apple solves this in the record layer by using Protobufs to serialize the records and then taking advantage of the field tags to rename or drop columns in the schema without having to rewrite the actual records. You could probably do something similar, though you would have to think about how to integrate with Ecto (which has no concept of field tags). Of course another benefit is compression of the field keys, which are much smaller as varints.

Please elaborate. Interested in learning more. Thanks!