It absolutely was not. Your comment was very restrained and respectful, and I very much appreciated it. You also kinda dodged the testing discussion, which I didn’t fully recall (it was a while ago!).
What I’m trying to get at, though, is that even if you had committed to a more aggressive defense of unit testing (as some did!) I would still take no issue with that, because that’s totally normal and reasonable. If I criticize something and then someone else defends it, that is normal. I expect that to happen. Of course I will still reply in disagreement, because that is what you do when you disagree.
That is the theme of my post here: resistance to change is normal and healthy, but also sometimes wrong. And it’s actually really hard to tell when it’s wrong. It’s hard to tell when it’s time to move on. It’s hard to understand why Postgres is not a great database in 2025. It’s hard to tell why unit tests suck for finding bugs until you actually have a need to hunt deep bugs, and most simply do not!
Your quote leaves out the next line, which was actually very important: I was trying to suggest that I could just as easily have done the same thing (defended TDD) if I didn’t just so happen to have known better at that moment in time. And I have only just discovered how great PBT is! I would imagine someone who’s been writing property tests for decades would look at me and shake their head. I’m not even good at writing them yet! I am trying, though.
I remember a while ago, on here, I suggested that it would be nice to have computed properties in Phoenix, and someone replied that they didn’t like them, and I was all bitter because I thought they would solve some problems I was having. Except after further educating myself on the topic I think that person was right, and I was wrong. But how could I have known? That’s just how it is.
Maybe my tone was slightly off (“woodwork” was a choice) but I liked the prose. Apologies.
I suppose we can now exchange roles. IMO with enough experience one can build an intuition that’s right 98% of the time on when resistance to change is wrong.
During my long and colorful consulting and contracting career Postgres has been quite fine for almost everything. No, really, and yes I said it. This is kind of the argument I truly hate but I am forced to admit it’s correct when we use the engineering’s golden rule i.e. use the right tool for the job: “We don’t need Elixir, our app is internal and we’ll never even have 30 concurrent users; Django / Rails / Express.JS is serving us fine”. Same with Postgres. Same with a lot of other stuff in the IT area… sadly.
The part I passionately will hate until my grave however is that this argument gets stretched like the proverbial cheese all the way to the surface of the Moon and is used as an universal defense that’s it’s never the time to move on.
I think you are misinterpreting what I said here. Perhaps this is in part because I wrote “skill issue”, which to be clear is a completely unserious zoomer meme. But also, I am not alluding to widespread technical incompetence as you seem to think, but rather a very narrow technical problem: how to test distributed systems.
I will save you the details, but the TLDR is that distributed systems have a reputation for being very hard to test because they are nondeterministic due to the network behavior. Normal (single-threaded) programs do not have this problem. So, if you try to test a distributed database using the same tools that you use for normal software, you end up with a system that is not very resilient.
The above has led to a reputation of unreliability for distributed databases.
However, it is actually very easy to fix this: you simply reclaim the deterministic behavior with a simulator. Once you’ve done this, you can go back to writing code as normal. It’s legit magic.
My point was that the general (developer) public is distrustful of distributed databases because systems developed using this superior testing methodology are, presently, somewhat uncommon. This is changing.
I am likely grasping at straws here but since you said that you just discovered PBT – which btw hugely surprised me, I’d expect somebody with your level of culture to have known about it for a while – then I’ll just mention it… TLA+?
It’s not that I have just discovered PBT exists, I have known about it for a while. It’s just that I have only just discovered why PBT exists, because I have only recently reached the point in my life where I’m writing code for which it is actually table stakes. Up until that point I had simply, you know… ignored it.
Do you see, now? I did not know any better!
For the record, simulation testing is a little different from classical PBT. For one, I can’t figure out how to do shrinking. And (borrowing from the Tigerbeetle team) I have actually been trying to stick to the term “fuzzing” instead because I don’t like acronyms very much. But it’s all the same stuff really, if you squint.
As for TLA+ I am aware of it but that’s all so far. I think formal methods are cool but they can’t prevent bugs in code, whereas simtesting actually can find bugs in the real implementation. Formal methods are probably still useful, though, and at some point I will take the time to learn TLA+.
It is fine! This comes up every time, and I will reiterate once again that Postgres Is Fine.
But, like, doesn’t it bother you that distributed consensus has been solved since 1988 and the answer to HA Postgres is “fail over manually”. Doesn’t it bother you that range-based autosharding has existed for 20 years and the answer for Postgres is “shard manually”. Doesn’t it bother you that the database is not properly checksummed and indexes can just randomly get corrupted and start returning the wrong data?
I really would like to move on! Unfortunately there aren’t really any good alternatives.
I’d work on drastically different things than I did all my career if I had the material choice.
That’s the tragedy of the commons most of us are facing.
I see what you mean: it’s time to move on from Postgres to something better. Alas, almost nobody can work on interesting stuff like this. Which is why a lot of us are rooting for you.
And of course I deeply appreciate it. Really: thank you, so much.
This is how change happens, at first: slowly, one person at a time. I do wish I could just write code and ignore everything else, but if I don’t spend time writing about things every so often then nobody will ever know or care. And it takes practice just like everything else, so I have to screw it up a few hundred times before I get good at it
BTW, I recall seeing a post recently mentioning your SQLite thing, and while I still don’t actually know what it is, please do keep at it! I think SQLite is interesting because unlike most RDBMS it actually takes the right side of all of the tradeoffs for a non-distributed database. And, shamefully, with Litestream or similar it’s probably more resilient than every other RDBMS lol.
I think Elixir does a good job of being both ergonomic and “correct” (assuming this means it delivers great performance without suffering from major drawbacks). It’s not without a few warts but they’re pretty minor overall. It’s a nice marriage of Ruby-inspired syntax with the benefits of the Beam so I don’t really see it as a tradeoff of one versus the other.
I don’t believe this to be true. I don’t think anyone wants to wade through a bunch of boilerplate or clunky syntax hence the popularity of languages like Ruby and Python. Syntax may become even more important going forward until a day where humans are completely out of loop.
As someone who has used it professionally, I think this is reasonable. Even Lamport himself proposed using model-checking - rather than the proof system - to gain confidence in the correctness of TLA specifications.
Having said that, if anyone does get a chance to learn TLA+, I highly recommend it. It does provide a rigorous way to think about state-machines, something we tend to do a lot of in Elixir.
My point was that syntax is mostly inconsequential. You might be arguing extremes here; I have seen enough Perl back in the day to understand why people said that “Perl is the only language that looks identical before and after RSA encryption” and I am obviously not advocating for Brainfuck. But by and large, syntax is mostly irrelevant. People do get used to it, integrate it as an automatic habit, and move on. Never understood why some people bashed f.ex. LISP or Algol or Ada as much. Meh. It’s syntax. Unless it’s 5x more verbose than would be rational then it’s OK.
I agree. That was one of my top secondary reasons to stick with it after the honeymoon period (original reason was actual parallelism and a solid runtime).
While I sometimes cringe at “don’t try to catch everything with case, we’ll add arms as needed and in response to production errors”, I can’t deny that this has helped me and many others, many times, to ship faster and with 98% confidence.
A Rust NIF with more features (current and planned) than exqlite, more or less, though in fairness they have some that I don’t just yet (extensions and encryption, which are also planned and quite easy to implement compared to a few other things that gave me a few more white hairs). Most prominent of the current ones are streaming with low-memory footprint and cancellation – don’t ask how many days did the latter one swallow; there’s a balance to be struck between raw speed in SQLite’s VM instruction executor, and ability to cancel before a statement executes. Boy, was that not fun at all to experiment with. But it was super interesting and informative at least.
And please don’t ask how much time it took me to mull over bundled vs. system-installed SQLite. (whispers: “Weeks”.) And the rabbit hole on why is CI failing my test suite when it always passed locally. TL;DR I was hammering SQLite with too many requests in the tests and it “intelligently” told me “out of memory” and boy just finding out that this does not mean it’s out of memory took me days. Sigh. Never claimed I am particularly good but let’s just say that my faith in humanity waned further the day I deciphered the problem and simply made a custom Mix task that ran each test file one by one, sequentially, and that fixed all occasional local and always-present CI problems.
Anyway.
btw since you are very interested in this stuff: Jepsen: NATS 2.12.1. Spotted it today on HN.
Yikes. I considered using persistent NATS for one hobby project next year as I am sick of Kafka but might as well just go for Redpanda (Kafka wire protocol compatible) and move on. Or persistent RabbitMQ queues. Who knows. It’s a Ph.D. activity to even find what you need in this space these days.
Any recommendation on a persistent queue with good guarantees without too much sysadmin / devops pain?
Something people often miss is that programmer productivity is the #1 priority for most software projects, which is why Ruby or Elixir could frequently be the best tool for a particular job, as long as the technical merits are “good enough”.
Of course, we could argue if a language that immediately seems more straightforward to read & write to newcomers because of its ergonomics will continue to provide productivity gains after someone has become an expert in it.
I think a strong case can be made for Ruby or Elixir enabling long-term productivity, though it’s probably weaker for larger Ruby projects.
I really don’t know much about formal methods, but I am quite sure they are useful as a first-pass to prove the correctness of a design and so on. In my case I’m not really designing protocols myself, so this is not as much of a factor. My consensus is VSR, my core architecture is FDB, and so on. I have been deviating but not that much. Since the correctness of the designs I’ve implemented is mostly established already, bugs in the implementation are my main concern.
It’s also worth noting that simulation isn’t enough either. There can always be bugs outside the simulator; that’s why the goal is to minimize the “not simulated” surface as much as possible. But there will always be blind spots, which is why Aphyr still managed to find correctness bugs in Tigerbeetle despite their extremely thorough and aggressive testing. As it turns out, there really is no substitute for paying a seasoned professional to spend every waking moment trying to break your database for 6 months.
I’m curious: what sorts of things have you used TLA for in a professional setting? Sounds very cool.
What’s the advantage of using Rust there? Or rather, how much (memory) bug surface is there in the NIF that Rust covers? I really have no idea what it takes to wrap SQLite in a NIF but I assume there’s a decent amount of glue needed, and that’s the Rust part?
Misleading error messages are always fun. The best part about trying to write zero-dependency code is that all of the stuff I used to be able to blame on other people is now my fault…
It is, and that’s what I was getting at yesterday! At this point I would not touch anything distributed or database-shaped unless the team behind it is serious about simulation testing and correctness. Anything less will be 100% loaded with these bugs. And if they show even the slightest indignation after being shown a correctness bug, run far away.
The only problem with FDB is accessibility. Their docs suck and they don’t care because they are a 3 trillion dollar company and it Works For Them. I can’t get upset, though, because they gave me the code and that’s all I needed!
I hope that Hobbes will solve the accessibility problem more than anything.
Less, or zero, crashes. I was very careful not to just fangirl over Rust when I resumed serious work on the project earlier this year. The point was to be ultra stable. Secondary, but almost as important goal is, to put a fence / shield in front of the plenty of undefined behaviours that SQLite would happily expose you to if you don’t manually enforce certain invariants and relationships between the function arguments. That latter part is still, and always will be, work in progress.
(RE: memory bugs, they are not bugs per se. It’s just that SQLite gives you no guarantees what will happen if you pass a certain combination of arguments to a function.)
Tertiary goal is to make everything type-safe and with a stable contract. SQLite PRAGMAs are notoriously shaky there.
Final goal: make sure you never put integer values in string columns and vice versa. SQLite mostly does not protect you from that. (Yes, even with STRICT tables.)
So TL;DR: zero runtime crashes and guardrails that guarantee consistent behaviour + type safety.
I love the idea of SQLite and after I finally add 1-2 more features to my library I’ll start using it (and thus SQLite with it) for a bunch of hobby projects. HOWEVER. It has a number of warts… but it’s still best-in-class in its niche. My library aims to put a concrete blanket over the very annoying defects of SQLite because at the end of day – I’ll say it again – most projects ever done on this planet do not need a separate DB server.
Yep, I am starting to learn that. I wanted to use NATS Jetstream so much! But I guess not, sigh. In the HN thread that’s linked to the Aphyr article that I pasted above, there are links to discussions where NATS authors don’t even understand the distributed problems they introduced. Ouch.
I do my best to read each and every new ElixirForum thread and carefully write down the links that interest me. I made sure to re-record this one, thank you. I know I’ll need an MQ in my next hobby projects (stuff that I like very much + it will be a portfolio) so this link is _very_welcome.
One relatively new offering is s2.dev. In the same HN thread (for the NATS Jestream Aphyr test) the S2 author says they do simulation testing.
As I said, many people on this forum are much quieter than they should be – trauma and core personal traits I suppose – but a lot of us are rooting for you.
Well, to be fair, the intersection of consensus protocols and durability is very dangerous and confusing and every major consensus implementation was broken when that paper was published. I would bet most still are, though I don’t actually know.
But that’s not what was going on here. This is not a case of “there was a very subtle bug in our consensus implementation”. Like, that happens. I would not throw shade for that.
These people literally were just not fsyncing to disk. This is not a case of incompetence but malice. They are doing it to look better on benchmarks. This nonsense is extremely common with database companies; most famously MongoDB, but it’s practically endemic. At this point I don’t trust database companies at all (with limited exceptions) because there is a massive financial incentive to lie incessantly. Jepsen is the only thing keeping them in check even a little bit.
As it turns out not writing your acknowledgements to disk does also break Raft and is indeed utterly negligent (seriously how can you possibly not know this), but that’s really just icing on the cake.
It was about 15 years ago, I was working on a bespoke RTOS for some custom silicon - 28 core Xtensa DSPs - for the Microsoft HoloLens AR headset. We used TLA to design the inter-core messaging and power mgmt algorithms.
I got around to checking out the s2 thing and it seems to be a closed source cloud thing built on top of another closed source cloud thing (s3). This has been a big trend with database-adjacent stuff lately and it’s something I’m not a big fan of for a couple of reasons.
The idea is that you build your fancy distributed system on top of an abstraction like S3 and allow it to solve all of the hard problems like consensus and durability for you. (OG S3 has poor consistency but there are new “versions” of it that are much better.) And, actually, this sounds like a pretty great idea on paper, and you might recognize it as being quite similar to my pitch for Hobbes or for FoundationDB.
But the difference is that you can run something like FDB on your own servers, whereas if you want S3 then you are locked in. I really don’t think that’s a great position to be in, personally. There are alternatives to S3 but none of them are really comparable as far as I can tell (especially for the new strongly consistent stuff). MinIO is the closest I’m aware of and my impression is that it’s a bit of a clown fiesta (plus they have been rugpulling).
The other problem with S3, and actually also with FDB, is that you cannot run fully-integrated simulation tests because they are external systems. These s2 people claim to be doing DST but unless Bezos handed them the source code (strong doubt) they are obviously mocking out S3. And when your system is entirely dependent on S3 for its guarantees and behavior, well, that means you’re mocking out your guarantees. Sus.
Actually, if Hobbes succeeds I think it will be the first system to ever solve that last point, because unlike FDB we can write the higher-level layers in the same language/runtime as Hobbes itself. And that only works because the BEAM is an advanced distributed runtime and Elixir is a friendly language. More on that later.
Ouch.
