Fl4m3Ph03n1x

Fl4m3Ph03n1x

GenStage and Flow resources?

I have been hearing about GenStage and Flow, a tool built upon it, for quite some time. I understand it allows us to program in a paradigm that uses stages and flows and has back-pressure included.

However I have a few issues when it comes to adopting them (or even considering it).

There is only the documentation

That’s it. Documentation alone is not enough for me, I don’t learn how to use a technology by simply glossing over the docs. It’s like learning Java by reading only the JavaDocs. Sure someone will be able to do it. I am not that someone. I am dumb as a brick :smiley:

Conferences

You may say there are some conferences, but they are completely biased towards using GenStage or Flow and how great they are. I want the complete picture. I want to know when it is a good idea to use it and when it’s not. Like Agents. I know when to use them and when not to use them.

Books

In all the books I have seen, there are no GenState chapters. Only GenServer. I get it that GenServer is a major building block, but when none of the books out there has a chapter about it, it really makes me wonder if we need GenStage at all.

Even today I asked about GenStage to @ElixirCasts and they didn’t reply - it makes me think this is not an interesting topic at all.


Do you guy know of any unbiased GenStage books or resources?

Showing Posts 1 to 7

shanesveller

shanesveller

Flow isn’t a successor, it’s a tool built on top of the primitives of GenStage. It says so at the top of the repository.

Did this actually need to be a new thread off of your process-mailbox post?

LostKobrakai

LostKobrakai

I don’t think you can compare coverage of GenServer and GenStage. One is a integral part of the language’s stdlib, while the other is a third-party package, which happens to be quite good for processing pipelines. It’s a common, but certainly not universal problem to face.

outlog

outlog

I only know of the talks

btw José is announcing Broadway tmrw

Announcing Broadway

Broadway is a new open source project by Plataformatec that aims to streamline data processing pipelines. It allows developers to consume data efficiently and concurrently from many systems, such as Amazon SQS, RabbitMQ, Google Cloud PubSub and others.

In this talk, we will discuss how the Broadway architecture connects multiple stages and producers, how it leverages GenStage to provide back-pressure, how we rely on OTP for fault-tolerance, and other features such as batching, rate-limiting, partitioning and more.

kokolegorille

kokolegorille

The chapter 5 (Demand-Driven Processing) of this book is a good example.

Fl4m3Ph03n1x

Fl4m3Ph03n1x OP

Fixed.

Yes. I don’t want GenStage and Flow to hijack the other thread. I want as many ideas as possible there, specially those not including GenStage or Flow. Placing this there would just obscure good resources in a thread that doesn’t even mention GenStage nor Flow in the title.

If it is that much common, than we should have books and resources about it. Phoenix is a framework, a 3rd party tool to Elixir that happens to be good for web projects, but you have a ton of books about it.

Me too =(

Fl4m3Ph03n1x

Fl4m3Ph03n1x OP

So, Flow will be obsolete now? Do are they meant to solve different problems within the real of pipelined workflows ?

Will have a look!

outlog

outlog

You may also be interested in Flow by Plataformatec. Both Broadway and Flow are built on top of GenStage. Flow is a more general and powerful abstraction than Broadway that focuses on data as a whole, providing features like aggregation, joins, windows, etc. Broadway aims to streamline data pipelines and focuses more on operational features, such as metrics, rate-limiting, and so on.
GitHub - elixir-broadway/broadway: Concurrent and multi-stage data ingestion and data processing with Elixir · GitHub

— All posts loaded —

Where Next? Top

Trending in Chat/Questions Top

Other Trending Topics Top

mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews