Episode 158 of Thinking Elixir. A challenge for many people learning Elixir is the change in the mental model. Coming from OOP to Functional changes how we reason about our applications and the old way doesn’t always map over cleanly. Also, Elixir is capable of doing a lot more than other languages and frameworks can and this means learning how to build a “normal” application may change. We talk with @sb8244 about his new book “From Ruby to Elixir” that aims to help people bridge that gap. While the title says “Ruby”, we discuss how many of the concepts apply to people coming from most OOP languages. We talk about different libraries to recommend to people coming from a traditional Rails stack and more!
6 Likes
Hey, thanks for keeping at it - I’m looking forward to new episodes every week and it can’t be easy to fit podcast production in anyone’s schedule and be consistent at producing it - well done!
So as I’ve listened to this episode at some point @sb8244 mentions he’s not convinced that scheduling jobs on Postgres can ever be as fast as doing it with Redis - and thus it won’t scale unless coupled with some real Postgres expertise.
How I’m thinking about this is first of all, for high traffic job scheduling one would need to have a deeper understanding of any of their subsystems - be that Postgres, Redis or something more exotic backing the queue. So my question is why is Redis expertise any easier to come up with than Postgres expertise? I’ve not worked with any Redis gurus in the past, but I’ve definitely seen some database ones.
Second, I feel like the time needed to execute the job should almost always be dwarfing the time needed to enqueue / dequeue the job. For example if most of the jobs are about sending emails, then sending those and retrying in the face of errors should most definitely require more time than inserting / updating a record in the database. So I don’t see why the faster enqueuing / dequeuing of jobs from Redis (if this claim is even accurate or if the difference is meaningful if it exists) would scale better, as it’s not the act of dequeuing that takes time but rather that of executing the job. It feels like one would be limited by their app server’s resources much earlier than by their Postgres/Redis server’s resources.
Lastly, the usual arguments for keeping one’s setup as simple as possible should apply here as well: If a website is already using a database, why complicate the setup by bringing in Redis too? Will the benefits of Redis make up for the complexities arising from managing additional infrastructure on production?
Anyway, not meant as an offense to Steve or anything (all the love for discussing your new book!) - just interested in digging deeper on people’s experiences on that front.
6 Likes
I believe it is significantly easier to run Redis than postgres—there’s just less moving parts. I’d say potentially an order of magnitude less complexity.
My experience comes from a fairly high throughput environment where even with strong postgres operational experience we hit walls with it. We ran Redis for multiple years with maybe 1 operational incident (self inflicted).
If we ran a postgres job queue on the same database as our main app—there’s a 0% chance it would’ve been viable. This gets amplified if you have 100+ worker pods executing 20-50 parallel jobs in parallel.
All of this said: I’d still recommend Oban 9/10 times despite this. I’m not sure the scale we were running at is typical and so I wouldn’t make recommendations solely on an outlier experience.
In the episode I brought it up as a potential gotcha with postgres backed systems and I’ll probably always maintain that position. That disclaimer doesn’t mean you shouldn’t use it and it certainly didn’t mean I’m advocating for Redis job system. If I were then I wouldn’t be shy about it and I’d be including it in the book.
5 Likes
