Hobbes - a low-level distributed database for the Elixir programming language

garrison · October 19, 2025, 9:05pm

Hobbes is a low-level distributed database for the Elixir programming language.

Hobbes provides a simple, safe, and scalable storage layer for Elixir applications. It’s designed to scale horizontally, replicate data across machines, and handle disk or machine failures automatically and without disruption.

Hobbes offers a transactional key/value API which can model a wide range of data structures and indexes. Transactions can span the entire keyspace of a cluster, even across machines, and provide the strongest possible consistency guarantees (strict serializability) by default.

Elixir apps and libraries can use Hobbes to build systems that achieve modern standards of consistency, durability, and availability. Building distributed systems to these standards is notoriously difficult. Hobbes is a tool designed to make it easy.

You can find the source code for Hobbes on Tangled:

sodapopcan · October 19, 2025, 10:35pm

You did it! Congrats!

That is all as this is way out of my wheelhouse (though have been creeping along with the interactions between you and jallum).

dimitarvp · October 20, 2025, 2:20pm

Congratulations on the release. I’ve been doing my best to follow the discussions but as this is pretty specialized material, at one point I gave up.

One thing that I would remark is that a few examples could help. F.ex. there was a wave of people on this forum asking for stateful actors, and there were discussions around how useful would that be at all (if an actor persists state that makes it crash on every restart after, that’s actually a worse state of affairs than before) but there are likely multitude of useful usages.

Want to give us an example of a few of those?

derek-zhou · October 20, 2025, 4:03pm

Have you looked at khepri:

There seems to be some overlap with your goals.

garrison · October 20, 2025, 7:25pm

I am indeed familiar with Khepri, and I think both it and Ra are important contributions in the direction of strong consistency on the BEAM. I have remarked before that it is quite strange there is no consensus primitive in OTP (e.g. a Paxos implementation), and Ra is literally that. BTW, Erlang is actually older than the first working consensus algorithms (Viewstamped Replication and Paxos).

Ra and Khepri are not, however, sufficient for my goals in particular.

MultiPaxos-style replicated log databases like Khepri are not meant to scale out and are designed to store a pretty small amount of data. Their tradeoffs also require them to store an unnecessary number of copies of the main dataset, which is fine for a small dataset but very bad at scale. Khepri also happens to be an in-memory database (the entire dataset is in RAM), which is not an architectural limitation but a tradeoff they’ve decided to take (which I’m sure is fine for their use-case).

Databases like this (see Zookeeper, etcd, Consul) are generally used as control planes rather than used to store the main dataset. The problem with this approach is that it means you actually have to build an entire distributed database. Something like Zookeeper is maybe 5% of an actual database.

Hobbes inherits from FoundationDB’s architecture. FDB is a reconfiguration system which is explicitly designed to store large datasets but provides a very open-ended data model. So FDB is maybe 80% of a database, but it solves nearly 100% of the “hard problems” of building a distributed database. Correctness is very hard, and FDB provides an abstraction which is correct and scales out of the box.

As I’ve mentioned in the past, I am interested in building tooling to replace things like Postgres, S3, and so on. I need an abstraction which can scale up to “real” datasets so that I don’t have to keep solving the same distributed problems over and over again. I want to solve them once, because they are very hard.

Hobbes is designed to provide strong consistency guarantees while storing several orders of magnitude more data than something like Khepri (and serving equivalently more traffic). Architecturally, the difference in complexity to meet that requirement is quite substantial, but that is what achieves my goals.

If you’re interested in the tradeoffs here, check out this excellent article which covers some of them.

garrison · October 20, 2025, 7:59pm

The lack of examples is very intentional, because there is no public API. A curious reader might find their way to the workloads/ directory and read some of the test clients there if they want to know what the private API looks like

I am very aware this is an extremely strange way to introduce a library, BTW. Building a database like this is a long journey, and doing it properly essentially means designing it to be tested. This project, like FDB, takes this to an absolute extreme. I remember hearing an FDB engineer remark that for the first two years there was not actually a database because they simply developed and tested everything in the sim. And indeed after well over a year of development Hobbes has never written a single byte to an actual disk.

It is very strange to write code this way; I’ve never done anything like this before. The entire codebase exists only to be fuzzed. It’s like a closed ecosystem: a digital terrarium.

You might think of “test driven development”, but amazingly I’ve come to the realization that unit tests are completely worthless. I had to stop writing them altogether. They don’t find any bugs, but they break constantly and have to be rewritten.

But what’s funny is when the day comes that Hobbes writes its first bytes to disk, the vast majority of the bugs will already be gone. It will be reliable from day one. Isn’t that weird?

Anyway, I have digressed a bit, but usage examples will come with the public API. There is a roadmap for what has to happen before that. There are no open questions at this point; I have a good idea of how to implement everything which is left. It’s only work, now.

I’m aware this means there are some who will click by, say “I have no idea what this is”, and leave. And that’s okay for now! It’s not ready for them yet.

garrison · October 20, 2025, 8:35pm

Hobbes is a database, and is definitely not a library for stateful actors. When I say “OTP primitive”, what I mean is that Hobbes resembles a very sophisticated persistent ETS table. Where you might use :ets to back an in-memory data structure, you would use Hobbes to back an on-disk data structure.

There is a bit more, though, because Hobbes is not simply “on-disk ETS”. (Actually we have DETS for that.) Hobbes is fault-tolerant and will scale to very large datasets across nodes. An ETS table is not fault tolerant (tied to one node) and cannot be (natively) sharded. Also, ETS tables do not have transactions at all.

And so this is why I say it’s like a new OTP primitive, because there has never been an OTP primitive which can do these things. The closest would be Mnesia, but it suffers from small shard sizes and poor consistency guarantees.

Maybe Hobbes could be used to build a library for stateful actors. But that’s just because it is a tool for persisting state in general. For example:

A job library like Oban could use Hobbes to persist jobs (they currently use Postgres)
An event-sourcing library like Commanded could use Hobbes to persist events (they also seem to use Postgres)
A distributed process registry could use Hobbes, taking advantage of its fault-tolerance and strong consistency

But the main reason I wrote Hobbes is to serve my own needs. As you know, I want something that can replace Postgres itself (and S3 and similar tools). Hobbes is essentially an abstraction layer which contains all of the hard problems associated with building a distributed database, so that I can reuse it to solve the “easy” problems each time. Much like Erlang solves the hard problems of distributed scheduling and message passing and so on.

I can then use Hobbes to build, say, a distributed filesystem, or a relational database.

Actually, the relational DB already has a name: it will be called Memex. And Memex will probably be the thing most will take interest in, but it does not yet exist.

Hobbes will always be the “pro tool” for those who want to get their hands dirty, but just not “rolling a database from scratch” dirty

jam · October 20, 2025, 10:00pm

I love the ambition of this project and congrats on this milestone. Excited to see how it evolves and what the public api ends up looking like for the eventual Postgres replacement.

Might be worth putting this message somewhere in the Readme along with whatever caveats are appropriate

dimitarvp · October 20, 2025, 11:35pm

Thanks for the examples, much appreciated.

I strongly resonate with this:

That’s practically most of my reason to still be a programmer. So you have at least one person who super strongly subscribes under this philosophy together with you. The IT area keeps chasing its own tail and solving the same problems over and over again. That is what is weird.

I could not resist to not digress a little bit myself, my apologies for that.

There are many unit tests that are too micro and one does indeed find themselves maintaining a contract they don’t know whether will be valid next week. I relate to that a lot.

But let me point out that fuzz / mutation / property / integration testing is also TDD of sorts. I am not here to start academic debates however, so let us not do that, I wanted to give you praise that you wanted to have a harness that makes sure that your expectations, whichever level they live at, are always met and checked. That is what truly counts.

The examples that caught my eye are the following:

(No idea about Commanded so not commenting on that.)

ETS is really good but I did find myself wanting to replace memcached and Redis with it and obviously could not because DETS and Mnesia are a separate industry at this point and I did not wanted to dig myself in that particular grave.

Oban – awesome!

Distributed process registry is something that I feel is still under-served. My last job involved Fly.io and man, the amount of times I’ve seen logs saying that the orchestrator can’t find a node is now burned in my brain and will not leave its apparently comfy spot soon.

Thanks for indulging the questions. I know that for a person like yourself working on the problem itself is much more important than serving commercial (and sometimes hobbyist) needs but many of us on the front lines are acutely aware of the suboptimal “state of the art” of a lot of tooling and dream of more. Thanks for moving that dream a little bit closer to reality.

sodapopcan · October 21, 2025, 2:30am

Despite my reading along with a lot of your convos around databases here, I mostly fall into this category. Thanks to the more recent posts between you and @dimitarvp I’ve got a clearer picture of exactly what Hobbes is. I do have some questions but I’m going to sit on those for now becauuuuse you have totally nerd snipped me with your TDD comment. The most succinct way I can put this is that I also generally dislike unit tests, but if writing them helps inform lower level design but then you throw away all those tests away once you move up a level, you’re still doing TDD! Even if (more accurately when) you end up completely rewriting all those lower level functions based on the higher level tests… yep, still by definition (with the original definition being extremely fluid) TDD

Really most of these small unit tests are just REPL driven development where instead of pressing up a bunch of times, you write your test in a file that is easy to run, edit, and save for as short or as long as you want.

garrison · October 21, 2025, 7:23pm

I have written a new introduction for the README, taking into account the above feedback. I don’t think I could have written the intro without this dialogue taking place first, so thanks to all of you for that.

garrison · October 21, 2025, 7:42pm

lol I see I managed to snipe both of you with this throwaway line.

Hobbes’s development is absolutely driven by testing. In fact, it’s test-driven to a frankly extreme degree. But it is not “Test Driven Development” in the popular sense. The first line for TDD on wikipedia literally explicitly mentions “unit tests” so I feel pretty justified that that’s the popular sense.

And I have written ExUnit tests as a means to develop features sometimes, but they are not unit tests. They are serving as stand-ins for a script, with the sole reason being that mix test is easier to type than mix run priv/scripts/whatever.exs. There is no resemblance to actual unit testing.

I did not have such an extreme opinion before working on this project, but unit tests are completely and utterly worthless. They are best used as a manual method of snapshot testing to catch regressions, but one would be better off doing actual snapshot testing there.

Once you are working on a program which is complex enough that you cannot exhaustively validate its correctness in your head, you need to start fuzzing. Every database-shaped codebase I have read (which is, at this point, many) has had fuzzers. I don’t know why people don’t talk about this more, but I guess it’s just something that you learn pretty quickly while working on such code.

Unit tests do not find bugs. They suck.

garrison · October 21, 2025, 7:56pm

Just keep in mind that I wrote that in the context of serving my needs, and what replaces Postgres for me may not necessarily replace it for you. You might like parts of Postgres that I dislike, for example SQL, and you certainly won’t be getting those parts from me!

But for Hobbes specifically, the idea is that you could use it to build your own database if you like. Query planning ain’t easy, though.

garrison · October 21, 2025, 8:21pm

Well good, now that we’re on the same page I can ruin your day by telling you I don’t think distributed process registries are a very good idea.

There is a specific architecture that I see people reaching to. Like you have a job queue, and you want to ensure only one process executes each job.

This is the wrong approach. No matter how hard you try it’s basically impossible to ensure there is actually only one process. I am not going to justify this statement, in part because clearly you have some idea of how much this sucks, but… it sucks. Don’t do it.

Instead, you can push the guarantees down to the datastore. Insert a job row, and then when it’s complete delete the job row and insert the result. We’ve discussed this before.

The problem is that in order to do this at scale you need a Very Big Multitenant Distributed Database or it gets very hard.

This project is, actually, that database.

I do not want a process registry to coordinate jobs for an external datastore. I want a datastore that enables me to not do that.