GenStage usecases and best practices

Elixir Noob here, I’m looking for advice about GenStage and some general guidelines when designing systems with Elixir.

At a high level, I’m looking to build a system where the end-user can modify/select each stage of a data pipeline, the start and end of said pipeline will always be the same, but the end user can add/activate a new stage that will do the additional computation.

Will using GenStage and the Flow module be a good fit for the usage I described? if so any examples or advice anyone can offer in regards to the implementation?

Cheers!

1 Like

Your description of the problem is very vague to say if GenStage or Flow is a good fit. I would first worry about writing the application and business domain. As an Elixir beginner, there are a couple things you would need to learn before dynamically managing GenStage pipelines, such as managing simple processes, and you can reevaluate later if you need the performance characteristics or the backpressure provided by GenStage.

To put it shortly, start with the simplest abstraction until you are familiar and you are sure you need more complex ones.

7 Likes

Thank you Jose, then I’ll do as you advice and go work on my basics before jumping into GenStage

1 Like

I.e. GenServer handles single events. GenStage handles a chain of events. Right?

Not really. GenStage servers are just GenServers as well, but they have some extra functionality build on top for handling events in a chain of producers and consumers (or producer/consumers which do both things), reducing the boilerplate needed to set those up. Flow is another layer of abstraction on top of that which handles common cases like faning out to multiple processes and reducing those back at a later stage.

All this could be done using plain GenServers as well, but you’d need to write a ton more code to wire things up. Also each of those servers, be it a genserver or genstage instance still works syncronously on it’s own events. It’s multiple of those servers, which makes things async.

2 Likes