jallum

jallum

Bedrock - a scaleable, distributed key-value database with better-than-ACID guarantees

Hey folks :waving_hand:

I’ve been building a distributed key-value store in Elixir/OTP called Bedrock. It implements FoundationDB’s architecture but with a twist - instead of running as separate database servers, the components run embedded within your application’s supervision tree.

It’s a key-value store with:

  • ACID transactions with strict serializability
  • MVCC with snapshot isolation (repeatable-read + read-your-writes)
  • Write-ahead logs for durability + storage servers for serving reads
  • Components that run as OTP processes inside your app
  • Recovery orchestration using supervision trees
  • Consensus over RAFT

The architecture follows FoundationDB’s model closely - resolvers, log servers, storage servers, commit proxies, sequencers, etc. The design is modular, so you can swap out or experiment with individual components without reworking the whole system. The interesting part is exploring what happens when these components run embedded in your application rather than as external services. Local reads can potentially happen at memory speed while still maintaining these strong consistency guarantees.

Current state: The core transaction pipeline is working. You can spin it up in a Livebook for experimentation or run it across a full cluster for real distributed transactions. It’s designed to scale from in-memory test instances, to local development through to production clusters. Recovery works when components fail, though there are still rough edges and optimization opportunities.

If anyone’s interested in distributed systems or has experience with FoundationDB, I’d love to get your thoughts on the approach. The modular design makes it pretty easy to experiment with different implementations if you want to try out ideas.

GitHub:

https://github.com/jallum/bedrock

Happy to answer questions or hear what you’d like to use it for. PRs are always welcome!

Most Liked

garrison

garrison

Hey, someone else has been doing this?! Looks like beautiful work! I’ll have to spend some time looking through the code later.

So glad to see FDB’s architectural influence growing. The design is just so good, and it maps really well onto the BEAM; probably because they FDB guys were somewhat inspired by Erlang/Actor model.

jallum

jallum

Bedrock 0.2.0

Major Features:

  • KeySelector API: Added comprehensive key selector support across storage, gateway, and repo layers with range query capabilities
  • New Olivine Storage Engine: New B+ tree(ish) storage implementation with advanced indexing and persistence
  • Range Reads: Full range-read support with key selectors, conflict detection and read-your-writes within transactions
  • Enhanced Transaction Processing: Refactored commit proxy finalization with synchronous sequencer notification and exactly-once reply guarantees

Architecture Improvements:

  • Reworked transaction tree balancing and processing
  • Improved resolver capabilities with enhanced telemetry and tracing
  • Director recovery system with capability-based retry mechanisms
  • Shale log consistency fixes and transaction stream enhancements

Developer Experience:

  • Updated documentation with transaction format guides and architecture deep-dives
  • Subspaces and fast tuple packing/unpacking
  • Enhanced example bank livebook with range query demonstrations
  • Comprehensive test coverage for new features

This release represents a significant evolution of Bedrock’s transactional capabilities with improved performance, reliability, and developer ergonomics.

Run in Livebook

jallum

jallum

Hey folks :wave:

Quick update on Bedrock! Just released v0.3.0-rc4 with some major API improvements based on community feedback.

What changed:

The transaction API got much cleaner. Instead of threading a transaction handle everywhere:

# Before - explicit transaction threading
transaction(fn tx ->
  put(tx, "user:123", user_data)
  get(tx, "config:app")
end)

# After - implicit context + Ecto-style returns
{:ok, config} = transact(fn ->
  put("user:123", user_data)
  config = get("config:app")
  {:ok, config}
end)

Subspaces are no more! The new Keyspace system replaces the old Subspace concept and features built-in encoding support:

# Works great with the directory layer
{:ok, app_dir} = Directory.create_or_open(["app"])
users = Keyspace.partition(app_dir, "users", key_encoding: Tuple, value_encoding: BERT)

# Familiar things are still here
alice_key = users |> Keyspace.pack({"alice", 42}) # still makes keys, but you get more control over *how*

# gets / puts / etc. use the Keyspace as context for understanding how to encode and decode your data
%{name: "John", age: 23} = Repo.get(users, 42)

:ok = Repo.put(users, 42, %{name: "John", age: 23})

# Range operations will decode and values for you, too, if you want:
Repo.get_range(users) |> Enum.to_list()
[
  {42, %{name: "John", age: 23}}
  {63, %{name: "Jane", age: 25}}
]

Also simplified conflict management, improved error handling, and streamlined the range operations.

Why this matters:

These changes make Bedrock much more ergonomic for daily use while keeping all the other goodness intact. The API now feels more “Elixir-native” rather than being a direct port of FoundationDB patterns.

The encoding system is particularly nice - you can plug in custom serialization for keys/values while the keyspace handles encoding automatically. Great for when you want structured data but it gets out of your way if want to manage binary keys and values directly. It’s made the class scheduling example very succinct.

This version focuses on the concrete improvements users will see, shows before/after code examples, and explains why the changes matter for real applications. The suggestions here have been phenomenal, and I’ve tried my best to integrate them all.

Always interested in more feedback, especially if you’ve worked with FoundationDB or other distributed stores!

Run in Livebook

Where Next?

Popular in Announcing Top

gabrielpoca
Hello everyone! I want to share with you something that I’m really proud of: https://stillstatic.io/ Still is a static site builder for...
New
josevalim
EDIT: since Ecto 3.0 final version is out, this post was amended to use the final versions in the instructions below. Hi everyone, We a...
New
kelvinst
Hey everyone! Well, we made this lib a while ago and now we decided to finally go out and public with it! It’s a tool for creating and m...
New
mbuhot
Leverage Open Api 3.0 (Swagger) to document, test, validate and explore your Plug and Phoenix APIs. Generate and serve a JSON Open API ...
New
michalmuskala
Another small library today. PersistentEts Hex: persistent_ets | Hex GitHub: GitHub - michalmuskala/persistent_ets · GitHub Ets table ...
New
benlime
LiveMotion enables high performance animations declared on the server and run on the client. As a follow up to my previous thread A libr...
New
OvermindDL1
Been making an MLElixir thing (not released yet…) for fun in spare time in the past day. I’m just trying to see how much I can get an ML...
132 14057 106
New
anshuman23
Hello all, I have been working on my proposed project called Tensorflex as part of Google Summer of Code 2018.. Tensorflex can be used f...
New
alisinabh
Hey everyone i’ve developed a library for Jalaali calendar for elixir which supports converting Gregorian dates to Jalaali and vice vers...
New
mattludwigs
Grizzly is a library for working with Z-Wave devices. Z-Wave is a low-frequency radio protocol for controlling smart home devices on a me...
New

Other popular topics Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement