Bedrock - a scaleable, distributed key-value database with better-than-ACID guarantees

Hey, someone else has been doing this?! Looks like beautiful work! I’ll have to spend some time looking through the code later.

So glad to see FDB’s architectural influence growing. The design is just so good, and it maps really well onto the BEAM; probably because they FDB guys were somewhat inspired by Erlang/Actor model.

Amazing work! Do you have a roadmap on things to work? I would gladly contribute for it, it’s something that has been on my mind for a good while

One really important thing that is missing here which in my opinion is the main feature for FDB being FDB (as well as tigger beetle and antithesis) is the simulation testing. From a glance it seems that nothing of the sort is implemented (correct me if I’m wrong), is something you have in mind? Having this system is pretty much the foundation of those databases and in the case of FDB was pretty much the first thing implemented before anything else

This looks very cool – congrats on the milestone release!

I ran through your Livebook and unit tests, and read some code, and everything worked as promised. Excited to see where this project goes – do you have an recommended starting points for an interested potential contributor like myself?

What are your thoughts on a storage engine? Is it possible to give the same Elixir treatment to FDB’s redwood?

Last thought, as the Layer guy: The cool thing about Data Layers is that it should be totally reasonable to swap out FDB with an API-compatible backend like this one, with a little glue code.

Yep, you’re right. No simulation testing at this time… and frankly, nowhere near enough testing period, unit or otherwise – it’s early days. I’ve been trying to think of ways to go about it, and I’ve not come up with anything i was happy with… I just don’t know enough about the internals of the BEAM to make it go. I’d be really interested to hear how you might go about it, if you have ideas!

Thanks for taking a look! When I read the fdb paper a few years ago, i thought exactly the same thing: “This would be so neatly expressed in Elixir/OTP.”

I made a bedrock channel on the elixir-lang slack, if you (or anyone else!) wants to ask questions or nerd out.

This looks very cool – congrats on the milestone release!

Thanks! I’m an admirer of your ecto-foundationdb project… It kind of pushed me into building this, actually.

I ran through your Livebook and unit tests, and read some code, and everything worked as promised. Excited to see where this project goes – do you have an recommended starting points for an interested potential contributor like myself?

I’m glad to hear it all worked!

Absolutely there are places to contribute. I’m working on a list of things in the “issues” on github, and I’m very open to more ideas. I have a bunch of write-ups in the “guides” section of the repo that breaks out how the various components interact and how processes (like recovery) are working currently. I’m actively working to improve the docs to make the project more accessible.

Things that could be fun:

• An ephemeral version of the log / storage servers (simple, backed by :ets) that could be used by end-users to run unit tests without needing to configure disk space, etc.
• Storage recovery. There’s no mechanism to replace/clone a completely failed storage instance.
• Data Distribution. There’s no mechanism to split or coalesce shards, and and :dets (which underpins the example storage engine I have in there now) is limited to ~2gb, making the system as a whole limited to ~2gb.
• Testing. It isn’t glamorous, but this is a db, and people are going to expect it to work. There’s a fair bit to do here. I plan to spend a bunch of time working on this to make sure that what’s already done works.
• Configuration. I can foresee some mix tasks to allow easy access to change the parameters and configuration of a running instance. It’d be pretty dope if you could start out with an ephemeral single-node system and configure it into a multi-node cluster without stopping it.

What are your thoughts on a storage engine? Is it possible to give the same Elixir treatment to FDB’s redwood?

The storage system is very pluggable. I’ve done some research into Redwood, and I think we can actually do better in some ways. The state of the art has advanced considerably in the db world since it was introduced. I’d like to try mixing some of the ideas behind Aerospike (in-memory indexes) and TiDB (Blob-backed storage with LSM tree compaction outside the critical path).

Another thought was that it might make sense to have multiple types of storage / log engines at play, at once. You could imagine a memory-only engine for high-speed reads. Another that persists data out on blob-stores like GCS for extreme durability. Another (like redwood) that would allow versioned reads going back further than 5s, to support snapshot reads that run much longer while remaining consistent.

Last thought, as the Layer guy: The cool thing about Data Layers is that it should be totally reasonable to swap out FDB with an API-compatible backend like this one, with a little glue code.

This was also my thinking. I could very much see adding higher-level Layers like sql (perhaps based on ecto_foundationdb), job queues like Apple’s QuiCK, etc. It would be wonderful if we could have these things all work within the same transactions – a very exciting prospect.

Exactly! I’ve been working on something fairly similar for a similar amount of time and it looks like we’re probably going to end up going public on here within a few weeks of each other (I’m still prepping), which is pretty hilarious! I will definitely give the codebase a full read and give some thoughts if you like. I’m confident between myself, you, and @jstimps we can achieve Elixir FDB supremacy

If I might offer one early point of feedback: as someone who naturally clicks directly to the “docs/architecture” section of any repo I’m finding the internals docs to be rather verbose (I assume LLMs are involved?). That style is okay for a README and such, but for the really technical docs I’d recommend seriously cutting down on the prose as the signal to noise ratio can suffer.

I’m here for it. Where’s this repo of yours?!?

Yeah, you’re not wrong. It’s just been me… and technical writing isn’t a super-power of mine. So, I’ve been relying on ai-tooling to help, and it could definitely do with some tightening up. With the basics now in place, I plan to spend some time working on that.

I have been looking into this, it’s definitely not easy. A storage engine benefits from a low-level language like C/Zig/Rust for a number of reasons. Erlang also apparently lacks direct I/O support, which I found pretty surprising (even Node supposedly supports O_DIRECT, though I honestly wonder if anyone has ever used it). But maybe the mandatory aligned buffers are a problem, I don’t know. I’ve been meaning to make a thread about that.

There are a couple of pure BEAM storage engines, CubDB in Elixir (which is not really suitable) and leveled, the backing store for modern OpenRiak. leveled is an LSM written in Erlang.

For the record, Apple seems to be spending quite a bit of dev time trying to replace Redwood with RocksDB if you pay attention to the commits, though of course in typical FDB fashion they have said absolutely nothing about why. I think Redwood might have been a Snowflake project and it doesn’t seem like they contribute anymore. Apple is probably storing considerably more data in FDB than Snowflake so if I had to guess I’d say they want to cut down SSD costs with an LSM as Btrees can be faster but are more wasteful (SSDs are consumables at scale).

Redwood is a pretty recent storage engine. It might actually be the most recent Btree storage engine out there, at least of those operating at any appreciable scale.

Do you have more info about this? I don’t know much about TiDB.

I think LSM compaction is generally outside the critical path, though, which is why you get into so much trouble when they fall behind. There is actually one exception I’m aware of: Tigerbeetle’s LSM deterministically interleaves compaction with writes, which is very clever.

Soon! Prepping!