Ecto_foundationdb - An Ecto Adapter for FoundationDB

Your Libraries & OS Mentoring Libraries
,
1

I’ve started development on an Ecto Adapter for FoundationDB: GitHub - ecto_foundationdb.

FoundationDB is a distributed database with ACID transactions. ( https://www.foundationdb.org/ ). The adapter is still very early stage, but some basic functionality works, and I’m interested in gathering some early feedback.

There are no published docs yet, but in addition to the README, some more documentation can be found in the Ecto.Adapters.FoundationDB module.

8 Likes
2

Announcing EctoFoundationDB 0.1.0!

EctoFoundationDB is an Ecto adapter for FoundationDB, a distributed key-value store that is designed to be scalable, fault-tolerant, and performant.

Quick Links:

Features:

  • CRUD plus indexes
  • Multi-tenancy
  • Automatic migrations
  • Custom indexes
  • FDB Transactions

Due to FoundationDB’s Layer Concept, EctoFoundationDB is a more than a wrapper. It has opinionated default behavior that is intended to fit the needs of modern web applications, and it also allows you to add structure to your data beyond the table. It does both of these things with ACID transactions, to ensure your entire data model is in a consistent state no matter what.

For example, maybe you want to put all your Users in a durable queue to process later. Or maybe you’re interested in implementing your own vector similarity search directly on top of your existing data model. Perhaps you’re intrigued by the sound of automatic schema migrations. Maybe you just need some very solid simple data storage with high availability.

EctoFoundationDB can help you do any of this.

For me, after managing various medium-to-large-scale SQL and NoSQL databases in production for 12 years and eventually deciding I’m more of a NoSQL guy, I simply wanted an Ecto adapter where I felt like I was at home.

Finally, thanks to @Schultzer and @warmwaffles for their open source adapters. I learned a lot from ecto_qlc and ecto_sqlite3, and you should definitely check them out!

6 Likes
3

Nice! I was looking at possibly building a FoundationDB adapter, but I haven’t used it before to really know what I was getting myself into.

1 Like
4

This looks awesome.

1 Like
5

EctoFoundationDB 0.2.0 is released (changelog)

There are 2 new features for writing fast transactions:

  • Pipelining: The technique of sending multiple queries to a database at the same time, and then receiving the results at a later time, still within the transaction. In doing so, you can avoid waiting for multiple network round trips. EctoFDB provides an async/await syntax on the Repo.
  • Upserts: Support for Ecto options :on_conflict and :conflict_target

For example, we can combine these features into the transaction below, which safely transfers 1 unit from Alice’s balance to Bob’s balance. With Pipelining and Upserts, there are 2 waits for data from the network (best case).

(Reminder: FoundationDB’s transactions are ACID and globally serializable)

def transfer_1_from_alice_to_bob(tenant) do
  Repo.transaction(fn ->
    a_future = Repo.async_get_by(User, name: "Alice")
    b_future = Repo.async_get_by(User, name: "Bob")

    # 1. wait (Alice and Bob pipelined)
    [alice, bob] = Repo.await([a_future, b_future])

    if alice.balance > 0 do
      # No wait here (because of `conflict_target: []`)
      Repo.insert(%User{alice | balance: alice.balance - 1}, conflict_target: [])
      Repo.insert(%User{bob | balance: bob.balance + 1}, conflict_target: [])
    else
      raise "Overdraft"
    end

  # 2. wait (transaction commit)
  end, prefix: tenant)
end

Compare with this logically equivalant transaction, implemented without pipelining nor upserts. It waits 5 times for data on the network.

def transfer_1_from_alice_to_bob_but_with_more_waiting(tenant) do
  Repo.transaction(fn ->
    # 1. wait
    alice = Repo.get_by(User, name: "Alice")

    # 2. wait
    bob = Repo.get_by(User, name: "Bob")

    if alice.balance > 0 do
      # 3. wait
      Repo.update(User.change_balance(alice, -1))

      # 4. wait
      Repo.update(User.change_balance(bob, 1))
    else
      raise "Overdraft"
    end

  # 5. wait (transaction commit)
  end, prefix: tenant)
end

Thanks for reading!

2 Likes