GenServer vs RabbitMQ/Kafka

Your inclinations seem correct to me. If you want guaranteed delivery/handling then you should use rabbitmq or kafka. I’ve used rabbitmq in the past (with events sent from a rails app and received in an elixir microservice) and it worked very well. In that case I used GenServers to handle sideeffects. So I had GenServers to receive messages from rabbitmq and then to make changes in redis and salesforce. As rabbitmq already provides backpressure control I didn’t bother with genstage but I believe there are some rabbitmq configurations where using genstage could make sense.

Sneaky answer, but RabbitMQ is erlang and therefore probably also OTP

But you’re surely correct in your assessment. If you want a persistant message queue a plain GenServer is probably a quite simple and not very featureful one, which misses the persistance to start with.

Right, for the most part, if I really want high throughput I would try to stay on Elixir land, like a chat or some soft of real-time thing.

But I would prefer to do what you are doing, use Elixir as the worker implementation but rely on those tools for the message passing.

hah yeah indeed OTP but with second level steroids included

This is my fear though, I feel that I don’t gain that much compared with RabbitMQ if I start including the edge cases.

You can gain a lot, like simplicity and easier deployment. With DETS one can solve the persistance issue. But if that works for your case is a whole different question. It’s always tradeoffs.

Also why is that a fear. Did you expect GenServers to give you more tools and you didn’t get them?

The fear comes form the unknown, for the most part so many things could go wrong like I messed the hot replacement, my server for whatever reason went down, I did a poor job in the implementation and somehow I had to shutdown everything, I need to scale the app and now I need to deal with multiple VMs

But again, unknown.

Right but if I keep adding more complexity to gain what RabbitMQ would give me from the getgo, is where I start questioning myself if I should keep going with this route.

That’s something only yourself or people on the project can answer. I just wanted to say there might be benefits over using a ready to be used solution, but those might be ones that don’t matter for your usecase.

GenServer vs RabbitMQ … both sides are for very different use cases. RabbitMQ is a messaging bus or a queue, see AMQPUseCase where else a GenServer is a processing unit.

For some applications it is natural to have RabbitMQ as the core with GenServer (or even Tasks) hanging off it. For other applications RabbitMQ is use to interact with non-BEAM subcomponents. This is especially true if you cross data-center boundaries. Shovel things into RabbitMQ and have RabbitMQ shovel it across.