Building something ad-hoc from various tools like GenStage, Stagger, Broadway, RabbitMQ, etc., would certainly work and could be an excellent learning opportunity. From a contrasting view, as some people have mentioned, you could do this with Oban starting now and scale into the 100/k range with little effort (many businesses are processing 10s of millions of Oban jobs daily).

Here are some things to consider (disclaimer, I’m entirely biased and not saying it’s the only way ):

• Horizontal scaling. Ingesting and handling that many events will require more than one system. Tools like GenStage are bound to a single node, meaning you’ll need a centralized queue to scale out. Oban solves this.
• Partitioning. You have events coming from distinct users without any pre-defined batching. To effectively de-dupe and process those events, you’ll need your consumers to process events in arbitrary groups. Oban’s partitioned global limits and chunks solve this.
• Fairness. Bursty or more active clients can drown out less active clients and prevent them from being processed in a timely manner. To combat that, you’ll want to rate-limit events by the client in some way. Oban’s partitioned rate-limiting solves this.
• Retries. Sometimes, event processing will fail, and you don’t want to lose the events you were working on. You’ll need a retry mechanism and the ability to figure out what’s wrong. Oban’s retries, error reporting, and telemetry integration solve this.
• Observability. What’s your throughput? Where are your bottlenecks? Are there errors causing this? Oban’s telemetry integration, logging, and the Web dashboard solve this.

Toss in pausing, scaling, testing, graceful shutdown, and the fact that you already have Postgres running, and Oban has a lot going for it.

The article One Million Jobs a Minute with Oban should cover the baseline.

Separately, watch out for a Scaling guide in the official docs in the next week or so.

Since there still seems to be some confusion, let me try my hand at clarifying:

Elixir is absolutely great, almost perfect, for the scenario you are outlining.

The problem however stems from your steep requirements:

1. Raw speed. if you want every single request that comes to your app to be served in a timely basis (5-30s), and when we’re looking at 6-7 digits of requests per second, this becomes difficult even for uber-fast languages like Rust (and with some tuning, Golang). Not to mention the load balancers you need to put in front of all this since a single node can only occupy as much as ~60k network ports (there are workarounds but… long topic).

2. Persistence. You can easily spawn a million Elixir processes each serving a single request (or pertaining to a single object or batch of objects) and it is practically the best runtime environment on the planet to handle this… BUT… if your VM node goes down, all these spawned tasks will disappear. If that’s undesirable then you should use Oban. And when persistence and atomicity get involved then you’re looking at drastically reduced throughput of requests (tasks) per second.

The part from the picture that I am still missing is this: are you supposed to receive normal HTTP requests and respond within, say, 5-30 seconds to each request? Or are your API endpoints simply supposed to ingest data coming from the outside (in the shape of requests), store them and eventually process them and send responses… somewhere else, not in a HTTP response?

I couldn’t get that part from your comments, and that detail can change the recommended solution quite a lot.