Bedrock - a scaleable, distributed key-value database with better-than-ACID guarantees

[jallum]

Hey folks

Quick update on Bedrock! Just released v0.3.0-rc4 with some major API improvements based on community feedback.

What changed:

The transaction API got much cleaner. Instead of threading a transaction handle everywhere:

# Before - explicit transaction threading
transaction(fn tx ->
  put(tx, "user:123", user_data)
  get(tx, "config:app")
end)

# After - implicit context + Ecto-style returns
{:ok, config} = transact(fn ->
  put("user:123", user_data)
  config = get("config:app")
  {:ok, config}
end)

Subspaces are no more! The new Keyspace system replaces the old Subspace concept and features built-in encoding support:

# Works great with the directory layer
{:ok, app_dir} = Directory.create_or_open(["app"])
users = Keyspace.partition(app_dir, "users", key_encoding: Tuple, value_encoding: BERT)

# Familiar things are still here
alice_key = users |> Keyspace.pack({"alice", 42}) # still makes keys, but you get more control over *how*

# gets / puts / etc. use the Keyspace as context for understanding how to encode and decode your data
%{name: "John", age: 23} = Repo.get(users, 42)

:ok = Repo.put(users, 42, %{name: "John", age: 23})

# Range operations will decode and values for you, too, if you want:
Repo.get_range(users) |> Enum.to_list()
[
  {42, %{name: "John", age: 23}}
  {63, %{name: "Jane", age: 25}}
]

Also simplified conflict management, improved error handling, and streamlined the range operations.

Why this matters:

These changes make Bedrock much more ergonomic for daily use while keeping all the other goodness intact. The API now feels more “Elixir-native” rather than being a direct port of FoundationDB patterns.

The encoding system is particularly nice - you can plug in custom serialization for keys/values while the keyspace handles encoding automatically. Great for when you want structured data but it gets out of your way if want to manage binary keys and values directly. It’s made the class scheduling example very succinct.

This version focuses on the concrete improvements users will see, shows before/after code examples, and explains why the changes matter for real applications. The suggestions here have been phenomenal, and I’ve tried my best to integrate them all.

Always interested in more feedback, especially if you’ve worked with FoundationDB or other distributed stores!

[jallum]

TIL: Erlang’s :dets is written entirely in Erlang. For whatever reason, I thought it was going to be written in native code. Nope.

[garrison]

@garrison

I have begun to doubt my own argument in favor of transact() here I think. Gonna have to spend more time on this one.

I spent more time on it and I have changed my mind.

The underlying cause of the double {:ok, {:error, "foo"}} situation is the coercion of throw/catch-style control flow back into case-style control flow. The reason for this is that RDBMS generally trigger a rollback on constraint violations by default so Ecto has to exist within that abstraction. This is the cause of all sorts of weird edge cases I had never thought deeply about. E.g. what if you catch a constraint with a Changeset? The transaction is still aborted but can continue by ignoring further operations. If you think about it this behavior is actually insane.

Where it gets interesting is with nested transactions. It turns out that throw/catch-style control flow is very useful here because if a constraint is violated you want to pop all the way out of the stack. Retrying only the nested part of a nested transaction makes no semantic sense because transactions are run on a snapshot which will simply violate the constraint again. So you need to retry from the beginning (at a new snapshot).

So why not “use the platform”? I don’t need a rollback() function if I can just raise. Then I can write my own retry loop around transaction() with a try/rescue. In order to write a unified retry loop this implies that the KV layer needs to raise on commit failures as well.

Putting it all together, I am now favoring a stripped-down transaction!() API which spits out its return value verbatim and raises on failure.

Repo.transaction!(fn ->
  Repo.put("foo", "bar")
  "foo"
end)
|> case do
  "foo" -> :ok
end

There could then be a transaction() which has the old (tuple wrapping) behavior but the raising version would be strongly favored for retry loops. I also considered whether the retry loop behavior belongs in the KV layer but, at least in my case, I don’t think so. How to retry is going to be intimately married to how constraints are implemented which is behavior that belongs in higher layers. In my case I intend to work with layers myself so I can always move the abstractions around if I turn out to be wrong about this. But I think of the KV layer as a “pro tool”.

For Bedrock maybe you are interested in exposing a more friendly API to users, in which case perhaps you would want to provide a retry loop out of the box.

Either way, I retract my argument in favor of transact(). My instinct was that lessons from Ecto apply here but I no longer believe that’s the case. The concept of “rollbacks” can be thrown away entirely.

[jstimps]

@garrison

considered whether the retry loop behavior belongs in the KV layer but, at least in my case, I don’t think so. How to retry is going to be intimately married to how constraints are implemented which is behavior that belongs in higher layers

I agree that the retry should be client side, at the layer if you will. But for what it’s worth, in the rare occasion I’ve needed a change in retry semantics at a higher level layer, I’ve made sure my invariant is represented by a set of keys in the DB and relied upon key conflict errors to drive my retries, adding explicit conflicts if necessary, but usually not.

In other words I’ve never personally found myself wanting a retry aside from the behavior that’s recommended for FDB clients by FDB API itself. Maybe there are other cases though. Just adding my experiences.

Regarding rollback, thank you for sharing your thoughts. I’ve never been able to apply rollback to my mental model of FDB style transactions, and appreciate you putting words to my intuition.

[garrison]

@jstimps

In other words I’ve never personally found myself wanting a retry aside from the behavior that’s recommended for FDB clients by FDB API itself.

The salient point here is that retries are something which are inherently tied to higher-level behavior. Some constraints might be retryable, like taking a lock, and some might not, like reserving a username, which is likely to remain reserved even if you retry. You might think of that latter case as having its retry loop in the human layer

I think by using exceptions I can then funnel all of the different failure cases (network failure, key conflict, constraint failure, arbitrary application failure) through the same code path. The loop can decide which errors should be retried and which should not. I think this may end up being semantic at the application level. Therefore I don’t think the KV layer is the best place to worry about such things. It can just raise. Like I said, “pro tool”.

@jstimps

I’ve made sure my invariant is represented by a set of keys in the DB and relied upon key conflict errors to drive my retries, adding explicit conflicts if necessary, but usually not.

I’m not sure if I understand what you mean here (an example might help).

But have you considered that you are allowing your application code to be shoved into an FDB-shaped (key conflict) hole when really the control should run the other way?

@jstimps

I’ve never been able to apply rollback to my mental model of FDB style transactions

I don’t even think it’s the write buffering that’s the problem. Rolling back a transaction on constraint failure just doesn’t even make sense. What if I don’t want to roll back? Am I supposed to read the row first and emulate the constraint myself? Constraints literally exist so I don’t have to do that! Because it doesn’t even work under READ COMMITTED! I suppose this is just how 50+ years of accumulated mistakes shape up.

BTW, buffering writes as FDB does is an old technique. In Andy Pavlo’s courses I recall the term “private workspace” being used (though that could also be on a server). I’m pretty sure Spanner used this approach as well (with a thick client). And many more throughout history I assume.

IIRC one of the Spanner papers specifically mentions they don’t have RYW. As if it’s a good thing, too, which I find quite funny.

[jallum]

@garrison

Putting it all together, I am now favoring a stripped-down transaction!() API which spits out its return value verbatim and raises on failure.

  Repo.transaction!(fn ->
    Repo.put("foo", "bar")
    "foo"
  end)
  |> case do
    "foo" -> :ok
  end

This is essentially the conclusion I’d come to and represents the semantics I’d presented originally.

@garrison

For Bedrock maybe you are interested in exposing a more friendly API to users, in which case perhaps you would want to provide a retry loop out of the box.

Such a retry-loop exists, currently. There are a class of errors from the storage servers (temporarily unavailable, aborts, etc.) that are automatically retried (with an optional limit) with backoff + jitter.

@garrison

Either way, I retract my argument in favor of transact(). My instinct was that lessons from Ecto apply here but I no longer believe that’s the case. The concept of “rollbacks” can be thrown away entirely.

I can see a case for transact / rollback, but it’s a convenience. In ecto, rollback(reason) does two things: 1. it uses an exception under the hood to bubble up from deep nesting, turning reason into {:error, reason} and 2. it wraps up some try-catch boilerplate that you’d need to have in place otherwise, everywhere. The way they’ve done it – it’s entirely opt-in, so if a user doesn’t like the idea of rollback/1, they don’t have to use it.

[garrison]

@jallum

This is essentially the conclusion I’d come to and represents the semantics I’d presented originally.

There is no failure handling in the livebook (that I saw) so I think when I read the code I simply assumed it worked like Ecto.transaction() and wrapped the result outside of the call. I did not read carefully enough to track the return types.

But yeah, you were right IMO, particularly here:

@jallum

but I came to think that trying to divine the will of the caller by picking apart the result is just kind of dicey

If the app really wants a “rollback” they can just raise themselves. To be clear, I still think most transactions should avoid this exception-style error handling. The prevalence of rollbacks within Ecto seems to come straight from mistakes made by RDBMS, which I did not fully appreciate before.

@jallum

The way they’ve done it – it’s entirely opt-in, so if a user doesn’t like the idea of rollback/1, they don’t have to use it.

The rollback() is intimately linked with the {:ok, {:ok, value}} error coercion. You could have the former without the latter, but then what is the point? It’s just a useless wrapper around raise.

I’m not sure I’d be comfortable forcing application devs to wrap all of their transactions in a try block, but I do feel comfortable forcing layer developers to do it (sorry @jstimps) because they have a better idea of what retry abstraction to use.

[garrison]

Arguably off-topic but I randomly saw this post on Bluesky and I just had to post it here

0mn1core (@0mn1core.bsky.social)

Also, it sounds like Bluesky is going to be using FDB! Great for us as they’re very developer-facing and it will help raise awareness about the architecture.