I think there’s a few really important points to solidify here:

The first is that even if you’re running a Task for each request, those requests are still going to be queued in some fashion on the BEAM while they wait to be scheduled and executed. If you start 1000 tasks and the beam crashes you lost potentially 1000 requests. So I’m not sure stateless vs. stateful is a good comparison to draw here, mostly because its not an accurate comparison. A potentially more appropriate decision is to decide if you’re going to send messages “at most once” or “at least once”. The current semantics you’ve drawn up in the crashing scenario are “at most once”. That might be fine for your use case but its meaningful to make that decision consciously.

A second, and maybe more important point, is that while its potentially useful to imagine that the downstream service has infinite resources and infinite scale out, the harsh reality is that neither of those claims are true. Services have limits, networks have limits, and both of them can experience faults. This is why it’s generally useful to implement batching and pooling. You minimize the overhead required to service the number of requests you have.

Saying that the batching and queueing makes it harder to scale your app just isn’t an accurate statement, especially when you consider the system as a whole. We can think of these 2 systems as a combination of queues. Requests are queued in the first service, sent to the downstream service, the downstream service needs to handle them. It can do some of this work in parallel but given that it’s a stable queue there will always be work in progress, waiting in a queue somewhere. If the downstream service has a slowdown for whatever reason then our total time spent in the system queue (the first service queue and the second service queue combined) increases. This will cause slowdown throughout the whole system and can easily overwhelm your upstream service. There are a bunch of ways of handling this (load shedding, back-pressure, etc.).

One of the ways we’ve had the most success is to use pools for our communication to downstream services and dynamically increase and decrease the number of works available in the pool based on the number of good responses were getting. If we start overwhelming the downstream service we dramatically lower the number of workers and start load-shedding (or return cached good responses) in the upstream service. This allows our downstream services time to heal before we start sending even more traffic their way.

All of these techniques are more about allowing for more overhead in your services and gracefully handling transient failures. While its totally possible (and maybe even reasonable) to spawn a Task per request, depending on your specific needs and guarantees around message delivery you may want to consider a more robust solution.

Maybe I should backtract to the original question here.

One reason why I introduced the pool in Elixir in Action was precisely to warn the readers about a potential overload when unlimited concurrency is used. So the message you should take from the book is that it’s definitely good to think about overload scenarios. However, I don’t mean to imply that pooling is the only, or the best option in all such cases.

Which brings me to my comment about batching. I made that comment from the standpoint of load control (which is what IMO pooling is also about). The reason why I introduced this approach to the discussion was to show that there are other ways of controlling the load.

I agree with others here that both approaches are effectively stateful. If BEAM or the underlying machine goes down, whatever you’re doing in-flight will be lost.

It is, however, true that with batching you end up with possibly larger crash effects. If the queueing process crashes, you might lose more in-flight data, compared to a single task. Likewise, if the processing of the batched items fails on the consumer side, you might end up with more failed requests. This is a trade-off of the batching approach, and something you need to account for when considering it.

Either way, I don’t see any general reasons why each technique (or any other load control technique) would prevent scaling (particular reasons might of course exist in concrete scenarios). The statefulness of the queue is local, so you can e.g. still have multiple batching queues spread across multiple machines, just like you can have multiple pools spread across multiple machines.

In any case, all my comments were made from the standpoint of the load control, not scaling. So pooling in the book is used to control the load (it’s also used for teaching purposes, as a fairly simple but realistic concurrent challenge). The batching is mentioned in this thread for the same reason.

Do you control both services?

If so, the first thing I’d personally consider is batching, because I sometimes find that sending X requests at once results in much less processing than X time sending one request, and that can really do wonders for performance/stability.

Other than that, pooling + queueing + shedding can be used to control the load, i.e. to make sure that the target service doesn’t get overloaded. More recently I tend to bypass popular pooling libraries (poolboy & friends), and instead use my own parent library which I find has the good balance of flexibility/simplicity for these kinds of challenges.