For persistent queues, I’m becoming a big fan of NATS Jetstream. It has nothing to do with Elixir btw. Before that, I was using Faktory a lot (which is based on Redis).

Jetstream is distributed and highly available. You can create both queues and streams.

The protocol to talk to Jetstream is extremely simple. Most Jetstream concepts are simple, but the feature set is huge, despite its simplicity.

Jetstream also comes with an amazingly robust pub/sub system.

Jetstream is much more simple than Kafka, but the feature set is greater.

Hi @artem ,
I am mostly just repeating things that were already mentioned by other people in this thread, but I hope to offer some points on which to decide what to choose based on your specific needs.

As you mentioned, you can definitely use Kafka and/or RabbitMQ from Elixir. RabbitMQ is actually written in Erlang, and runs on the same VM as Elixir. That said, it very much depends on whether you only need scalable and concurrent execution of tasks, or if you need to persist your jobs on disk to recover from a restart of a worker instance without loosing messages.

If you do not need to persist tasks on disk, Elixir/OTP has already very good facilities for managing concurrent tasks, and to scale to a large number of concurrent requests, without reaching outside of the language ecosystem. That’s because OTP relies on exchanging messages asynchronously across lightweight processes. This is really one of the use cases that Erlang is built for. One thing to be aware is that GenServer will not automatically provide a back pressure mechanism: it’s possible to do that, but does not come by default. There are some libraries, such as the already mentioned gen_stage, that facilitate the implementation of pipelines to process tasks in parallel, with back pressure, throttling, and more.

If you instead want to persist those jobs to disk, to recover from a restart without loosing them, I’d consider something like RabbitMQ (with durable queues enabled) or like Oban (based on Postgres), depending on your specific needs.

I have the following book in my backlog, and a general idea of how things can be done in Elixir:

I would tackle it by using GenStage. It has concept of Producer and Consumers, which helps with consumption of data in batched fashion. If the machine learning Consumer gets overwhelmed, it can ask for less work from Producer.. (i.e. Backpressure)

If requirements get more complicated, then we can go for Broadway.

For persistence across restarts, I guess Oban will do the trick.