I have been hearing about GenStage and Flow, a tool built upon it, for quite some time. I understand it allows us to program in a paradigm that uses stages and flows and has back-pressure included.
However I have a few issues when it comes to adopting them (or even considering it).
There is only the documentation
That’s it. Documentation alone is not enough for me, I don’t learn how to use a technology by simply glossing over the docs. It’s like learning Java by reading only the JavaDocs. Sure someone will be able to do it. I am not that someone. I am dumb as a brick
Conferences
You may say there are some conferences, but they are completely biased towards using GenStage or Flow and how great they are. I want the complete picture. I want to know when it is a good idea to use it and when it’s not. Like Agents. I know when to use them and when not to use them.
Books
In all the books I have seen, there are no GenState chapters. Only GenServer. I get it that GenServer is a major building block, but when none of the books out there has a chapter about it, it really makes me wonder if we need GenStage at all.
Even today I asked about GenStage to @ElixirCasts and they didn’t reply - it makes me think this is not an interesting topic at all.
Do you guy know of any unbiased GenStage books or resources?
I don’t think you can compare coverage of GenServer and GenStage. One is a integral part of the language’s stdlib, while the other is a third-party package, which happens to be quite good for processing pipelines. It’s a common, but certainly not universal problem to face.
Broadway is a new open source project by Plataformatec that aims to streamline data processing pipelines. It allows developers to consume data efficiently and concurrently from many systems, such as Amazon SQS, RabbitMQ, Google Cloud PubSub and others.
In this talk, we will discuss how the Broadway architecture connects multiple stages and producers, how it leverages GenStage to provide back-pressure, how we rely on OTP for fault-tolerance, and other features such as batching, rate-limiting, partitioning and more.
Yes. I don’t want GenStage and Flow to hijack the other thread. I want as many ideas as possible there, specially those not including GenStage or Flow. Placing this there would just obscure good resources in a thread that doesn’t even mention GenStage nor Flow in the title.
If it is that much common, than we should have books and resources about it. Phoenix is a framework, a 3rd party tool to Elixir that happens to be good for web projects, but you have a ton of books about it.
