Honestly, not eager to see it. Since all major DB’s are written i C / C++, what advantage will the new one built in Elixir be ?! Performance wise ..
Mnesia can beat postgres with both hands tied on its back for the workloads it designed for, latency and the laws of physics dictates certain limitations.
3 Likes
What do you mean by “database” and by “major”? Elasticsearch for example is a database, has a very broad adoption for certain kinds of workloads and therefore would seem to fit a definition of “major”. And it’s not written in C or C++ but rather Java. There are other examples of server applications which chiefly manage data and access to that data and have significant usage which are also not written in C or C++.
There are many kinds of data and data access needs, not all of which fit neatly in the model popular relational databases work well in. The volume of data, the structure that data, sensitivities to latency, data access methodologies, etc. usually force you to make decisions about what capabilities you prioritize over others. Different database models, systems, and implementations give you different trade-offs and have value in that you can choose which capabilities matter to your specific application.
So yeah, if you just want to make PostgreSQL in Elixir, I don’t recommend it. But if you are selecting for a certain set of trade-offs where having an in-BEAM database makes sense for some workloads, I don’t see the problem with pursuing such a project. I certainly won’t judge the effort for not fitting my preconceived notions of what a database is.
3 Likes
This is the third time this has come up in this thread and I dodged the question the last two times so I may as well answer it properly this time. But I will first reiterate that I am not trying to claim Elixir is somehow the best language in which to implement a database. Rather, I wanted to write a database and I quite like Elixir, so it was the natural choice for me. And this is of course the Elixir Forum, a good place to discuss such things.
But anyway, there are arguments that can be made in Elixir’s favor here. First of all, we should avoid reductive statements about performance. Things are never so simple. When designing a distributed database there are a number of different subsystems to implement, each of which have their own performance characteristics. A storage engine, for example, might benefit greatly from low-level control over exactly how memory is allocated.
But, on the other hand, a lot of the code for a distributed database is business logic (which has very strong correctness requirements but performance is less important) and various forms of message passing. The thing about the BEAM is it’s really, really good at passing messages around. They’ve been optimizing that one use-case for like 40 straight years.
So if you want to implement a database in C++, you have to design an actor model/message passing framework yourself. This is, in fact, what the FDB team did 15+ years ago. And what they ended up with is the purest embodiment of the “poorly specified implementation of half of Erlang” snowclone you can imagine. All the good stuff like preemptive scheduling, process isolation, error handling, all of OTP. It’s all missing. I mean seriously, go and try to actually read the FDB source code. It’s not exactly beautiful.
Put simply, if you were going to implement a distributed database from scratch in C, properly, you would end up re-inventing all of Erlang/OTP/BEAM first. And then you have to catch up to 40 years of their performance improvements. I don’t know about you, but I don’t think I can solo that one lol.
Now there are parts of building a database that are performance-sensitive in such a way that they are slow when written in Elixir. Maybe not as slow as you think; I need to do more benchmarking but I think my concurrency control is within 50% of the performance quoted in the FDB paper, and all I did was naively implement it on top of ETS.
But for those slow parts, we have NIFs. Down the road I can just swap the slow Elixir parts out for C or Zig or whatever if I want to. The BEAM was designed to do this!
Performance aside, though, the advantage for Elixir really comes down to extensibility. This has come up a couple times up-thread WRT “layers”, but FDB was always meant to be less of a database and more of a “tool for building databases”. I think that paradigm could be considerably more powerful in Elixir, as it allows you to write the layers with a level of integration that is hard to achieve with something like FDB. Perf is not enough.
7 Likes
Interesting. Looking forward 
.
That was a detailed answer, thank you and good luck.
2 Likes
Recognizing that I have some Rust bias, I would say that only Rust has a good shot at this – I mean replacing C/C++ for writing stuff like this from scratch.
I would even think a future Rust-based OTP would just be 100% Rust with a few scripting languages bolted on to manage the runtime engine better… syntaxes of some well-known niche languages we know as Erlang, Elixir, LFE, Gleam etc. 
That’s completely besides your point, but I just wanted to chime in right there in your comment.
For the record, I too agree that databases, especially distributed ones, should jump with joy when they get to the performance problems. That would mean they have already solved some of the hardest computer science (and maybe even mathematical) problems stemming from the difficulties of distribution and the potential to lose connection at any point in time.
2 Likes
Go has also seen a lot of success in this area. CockroachDB and TiDB are both Go.
Rust is obviously also a popular choice, though. Notably the new Aurora DSQL, which has an unbundled/service-style architecture quite reminiscent of FDB in some ways, is written in Rust. For the record that article is super bizarre, they essentially claim that rewriting the DB in Rust magically fixed their architectural problems which I strongly doubt. But it’s marketing material so there was probably just some miscommunication/oversimplification I guess.
Personally as someone who doesn’t know Rust I have to say I find it very hard to read. On the other hand, I’ve read through some decent chunks of Tigerbeetle’s Zig source code and I found that very pleasant and readable even though I don’t know Zig either. So on those grounds alone I’m much more interested in the latter.
The other thing that bugs me about Rust is it seems totally captured by big tech, which I don’t love. Not as badly as Go, of course.
Lmao tell me about it. Good testing is absolutely critical as it is 100% impossible to write bug-free code in this space. You essentially just have to assume everything you write is broken (because it is) and come up with ways to test it, which as it turns out takes just as much work as writing the code to begin with (if not more).
Syntax and ease-of-reading-when-not-an-expert both matter exactly zero, let’s be real. I like Rust because it solves real problems I’ve suffered through all of my career, and even before that when I was a dumb cocky teenager thinking he figured out computers back in 1996 in a backwater Eastern European town. Big topic so not going to hijack your thread over it, just TL;DR me and a ton of other people like Rust despite its syntax (it’s not terrible but could’ve been better; I like Go’s better).
Huh? Rust has enough critical mass behind it that any attempt at a rug pull by any big player will instantly have exactly the opposite effect of what they’d aim for, namely almost everyone will distance themselves from the said big player and Rust will continue just fine.
It has gotten to the point of being a commodity. I wouldn’t worry one bit about malicious big players.
Au contraire, it could very easily be true but they haven’t gotten to the architectural problems that they will have with Rust. I prefer those but it has to be said they’re indeed more than zero.
1 Like
It would have mattered in that if Tigerbeetle was written in Rust I’m not sure I would have actually bothered to read the code in question and if I had I’m not sure if I would have understood it as well. I say this really more in praise of Zig than as a slight against Rust. Also, C++ is definitely still worse.
I still think what Rust is is very cool, though. I hope in the future there can be more languages like Rust but maybe with a somewhat more constrained syntax 
I’ll probably end up learning it eventually anyway for practical reasons.
I definitely don’t think they’re at risk of a rugpull, I was really just talking about vibes I guess. Elixir has much more of an “independent community” vibe to me, for example, even though it is used by big companies. I quite like that.
I was referring to an architectural problem they describe in the article (to do with scaling a partitioned WAL). They describe the problem and then instead of listing actual architectural solutions they just pivot to “lol we rewrote it in Rust no GC very fast” and never mention it again.
Presumably in reality they also fixed the bug in their architecture (you have to partition the log on the way in), but I guess we’ll never know!
1 Like
