I’m expanding my understanding of genstage and how to build systems that are more resilient to overload. The GenStage docs demonstrate use of the Erlang :queue module… however, I am not clear on why that is being demonstrated. The :queue module seems to offer some convenience functions for working with lists, but it also has some inconveniences. E.g. it can’t work with a range like 1..5 – enumerables need to be converted into explicit lists, and certain operations like split fail if the queue is too small, so its use requires some “defensive coding”. Under the hood, an Erlang :queue is just a list, so I would expect the performance characteristics to be pretty much the same, and I would expect both to be equally susceptible to being over-stuffed.

I tried 2 variants of the same GenStage producer just to get a bit more familiar with :queue, and I think the 2 variants are functionally equivalent (although, the “defensive coding” required to use :queue makes me favor the simple list implementation).

Perhaps the conversation should be more about identifying the inputs to your system that could be “3rd rails” and overload it… A simple list (or a :queue) is acceptable for buffering demand or cases where the data stored in-memory is expected to remain tangibly small, but not when the input might be huge and unbounded. Anyhow, using :queue seems to be a distraction.

Thoughts?