Question: Multiple producers with a single consumer, is there a way to ensure that the demand is spreaded across all producers ? to be more detailed, we have the following stages pipeline: [A] - [B] - [C] Stage A … a variable number of instances, processing incoming data Stage B … a single inst…

[image] odix67: I would like to implement stage [A] as a producer, but didn’t find any hint the GenStage source where the consumer may spread the demand across multiple producers. You can control the demand from the producer but you can control how often you ask from consumers (B and C) by im…

@josevalim thx for the fast (very fast ;-)) reply, it wasn’t very clear to me, but now it is, I’ve overseen that the handle_subscribe callback is called on both sides, consumer and producer. thank you and best regards odi

But in the case stage B is set to :manual, I see no way for stage B to determine if there exists demands from stage C. In the case stage C is busy, no demand, stage B shall stop asking stage A (there could be more then 1, which shall be asked round-robin alike) for events to prevent buffer overflow.…

Is there a way to get multiple producers, though? I have a project where a producer connects to an external websocket API and that API streams more data than a single producer can handle. I’m manually creating separate producers for different endpoints of the API, but it’s a bit cumbersome and I’m…

I think so… maybe this can help [image] GenStage with Multiple Producers with ordering Questions / Help I’m trying to build a pipeline with GenStage and have a situation where my producers are a bottleneck. I’d like to run multiple producers and have found several exampl…

That one appears to be about ordering events produced by different producers. My question is making sure the producer can handle the volume of data streaming in from the external API. (Ideally via running concurrent copies of the same producer, if that is feasible)

Oh sorry… anyway I have done a genstage pipelines with multiple producers using a queue as input.

How did you set up the multiple producers and control the number of them?

I have an old repo where I made some tests with Genstage 2 years ago… [image] kokolegorille/gs_pipe GenStage pipeline demo in Elixir 1.7.4. Contribute to kokolegorille/gs_pipe development by creating an account on GitHub. It might not be up to date, but I reme…

GenStage: spread demand over multiple producers

Questions & Help Questions

kokolegorille April 3, 2021, 9:34pm 6

I think so… maybe this can help