Most importantly, stability/reliability and performance/speed.
I experienced no end of problems with MySQL, and, touch wood, to date, zero problems with Postgres.
+1
I think irrespective of which language a new DB was built on there really would need to be a compelling case to get people to switch from something like Postgres - what usp could an Elixir DB have?
I didn’t realize Riak couldn’t be embedded. Seems like wasted potential; it would be pretty cool to have in-BEAM even just for unit tests/development.
I guess back in the day they were targeting the wider market so they never bothered?
a writeup for Any way to embed riak_kv? · OpenRiak · Discussion #23 · GitHub would be enough for me
I am well aware of the cap theorem lol. Was this a non-sequitur?
Free as in freedom.
I meant that one more as a success criterion than a “want”. Proprietary databases are essentially worthless to me, and VC funded “open source” databases have a habit of lying to their users/customers and turning into proprietary databases as of late. I think in order to be successful the developers of such a database will need to strongly signal their genuine belief in free software instead of using it as a carrot to trap customers.
I guess I should have been more clear.
Well you know what they say, mo engineers mo problems.
Strong consistency
Partition tolerance
High availability
These three are incompatible. Pick two
Free
Yeah, right, you want to have a database program with first-in-class features and you want to have it for free. Well, I mean, you can want that, sure, but no matter how much you want that, it sounds like a several years of work for like 5-ish top level engineers with very broad experience.
After some thought I’m going to add a few more points which I had elided from the OP. These are a little more specific.
Sum types. Sometimes it can be useful to define fields (columns) which can be one of several types. This is something which needs to be used sparingly but which would be very useful in certain situations.
Polymorphic relations. This comes up a lot on here with Ecto (there is literally a new thread today lol). SQL makes this really hard. But if you implement foreign relations as types then you get polymorphism from sum types. This could be a nice DX improvement.
Incrementalization (live queries). Realtime functionality is a selling point for Elixir (particularly LiveView). There are ways to hack this together with PubSub or WAL parsing (with Postgres), but by designing this into the database from the start you can ensure a consistent realtime snapshot across queries. This would, essentially, be the “holy grail”.
I think the third is obviously useful but I would be interested to hear dissent on the first two as I’m less confident there.
Also, I have moved this thread out of the ecosystem category as I don’t see any reason for it to be hidden.
The API is very good but they made a couple of mistakes. A big one is that they defined a standard encoding for keys (tuple layer) but not for values. Binary keys/values are a bad abstraction, which they clearly recognized in the key case but not the value case. As a result Apple’s own record layer uses Protobufs.
There are two problems with this. First it hurts interop between potential layers, which is unfortunate. But second it makes it very hard to do predicate pushdown, which is very important for a distributed database. You can’t push a filter down to the storage layer if the storage layer has no idea what it stores!
And the tuple encodings are not suitable for values because they have to be parsed in full to be decoded. Which is necessary for sorting and fine for keys (which are small), but would be very bad for row values as you have to decode the entire thing just to read one column out of the middle. What you really want is a header with the value offsets up-front.
The solution, I think, is to define a record encoding to go along with the key encodings. I think it should (logically) be an ordered map integer => value where values are typed. Then you define a header up front with a format like type, id, offset, length?. Similar to protobuf etc.
Integer field ids are sufficient, schemas and so on can be managed by layers, either hard-coded or dynamic (you’re familiar with this!) schema management depending on the use-case.
I know where you’re coming from here, but the relational model necessitates some amount of query planning to satisfy data independence. We have discussed this before, though at the time I don’t think I realized this was spelled out so explicitly in the original Codd paper.
I think the planner can be kept fairly simple, though. I would argue even your ecto_fdb index selection logic constitutes a degenerate query planner (though it does not achieve independence).
Even more important is ensuring some sort of escape hatch. I don’t mind the Postgres planner 90% of the time, but then every so often it just refuses to use the indexes that I know are better and that just drives me insane.
It’s so funny reading a post from someone who actually knows FDB well lol. You know exactly where the pain points are. At the end of the day Apple clearly has no interest in maintaining it as an open project and the codebase is just way too onerous to fork. But full credit to them for releasing it at all, they didn’t have to.
Indeed, Elixir has some of the best docs out there so an Elixir database should continue that trend.
Unfortunately distributed databases with good architecture docs are quite rare (I have read like all of them at this point). CockroachDB has an okay design doc and some decent RFCs if you go looking. The best I’ve seen are probably Tigerbeetle’s, there is a lot to learn from their architecture WRT safety. FDB’s docs are a fragmented disaster (but still certainly not the worst).
One really gets the impression that vendors see architecture docs as a moat and therefore have little incentive to maintain them. One of the most egregious I saw was Neon (disaggregated Postgres), which boasted about being “open source” but had no documentation at all that I could find for how to self-host it. And now they’ve been acquired, so do you think those docs will ever be written? Yeah right.
Exactly , it’s a language, but how the query planner decides to interpret and execute that, is sole on the query planner. And that is exactly why you would not need any other languages regardless of your data is
relational or not.
To take it a step further, there is nothing stopping us from querying mnesia, ets or dets with SQL. In fact this has already been done with Ecto.Query.
I’m sure it comes as no surprise that I agree with your list. 
I would be eager to use, and contribute to, a BEAM native DB with a low level API compatible with FDB’s core feature set (ordered keyset, get range, clear range, transactions, mvcc). A high level relational API is where many people would live, but the direct storage engine access is so powerful.
The FDB team never got the chance to implement all the Data Layers on top of their storage engine because of the acquisition. It would be exciting to have an ecosystem where those ideas could carry on.
Here’s a few things I would add to a wishlist