Worth mentioning that my knowledge is not that deep, so excuse my ignorance.
I have been questioning why would I want to do everything just with OTP; GenServer in particular (Low-Level API) because in the ideal scenario my server never goes down and I wouldn’t lose any message or any information stored in memory but that is the ideal scenario and I don’t want to bet on it if I can avoid it by simply change my architecture, or at least mitigate the risks.
Or having more than one VM and now I need to deal with replication …
But the more questions I ask the less I know how to answer them and the simple answer to most of them is using RabbitMQ or Kafka since they have built-in fault-tolerance and recovering mechanisms that I would need to implement myself if I really want to cover my ass and make sure that everything will be fine even in the worst scenario.
For the most part, what I truly care about it is having this message passing and distributed architecture but I couldn’t say that I have done anything in Elixir that would require that level of abstraction yet.
But before I get to this rabbit hole I would like to hear some of your thoughts on this topic.
Why wouldn’t I use GenServers and do such of low-level things when I could simply use RabbitMQ or Kafka?
Wouldn’t be simple to send some messages to RabbitMQ and focus on programming the workers?
An example of this would be sending emails, or notifications of some sort.
Your inclinations seem correct to me. If you want guaranteed delivery/handling then you should use rabbitmq or kafka. I’ve used rabbitmq in the past (with events sent from a rails app and received in an elixir microservice) and it worked very well. In that case I used GenServers to handle sideeffects. So I had GenServers to receive messages from rabbitmq and then to make changes in redis and salesforce. As rabbitmq already provides backpressure control I didn’t bother with genstage but I believe there are some rabbitmq configurations where using genstage could make sense.
Sneaky answer, but RabbitMQ is erlang and therefore probably also OTP
But you’re surely correct in your assessment. If you want a persistant message queue a plain GenServer is probably a quite simple and not very featureful one, which misses the persistance to start with.
You can gain a lot, like simplicity and easier deployment. With DETS one can solve the persistance issue. But if that works for your case is a whole different question. It’s always tradeoffs.
Also why is that a fear. Did you expect GenServers to give you more tools and you didn’t get them?
The fear comes form the unknown, for the most part so many things could go wrong like I messed the hot replacement, my server for whatever reason went down, I did a poor job in the implementation and somehow I had to shutdown everything, I need to scale the app and now I need to deal with multiple VMs
But again, unknown.
Right but if I keep adding more complexity to gain what RabbitMQ would give me from the getgo, is where I start questioning myself if I should keep going with this route.
That’s something only yourself or people on the project can answer. I just wanted to say there might be benefits over using a ready to be used solution, but those might be ones that don’t matter for your usecase.
GenServer vs RabbitMQ … both sides are for very different use cases. RabbitMQ is a messaging bus or a queue, see AMQPUseCase where else a GenServer is a processing unit.
For some applications it is natural to have RabbitMQ as the core with GenServer (or even Tasks) hanging off it. For other applications RabbitMQ is use to interact with non-BEAM subcomponents. This is especially true if you cross data-center boundaries. Shovel things into RabbitMQ and have RabbitMQ shovel it across.
