How to turn off/on a broadway pipeline?

Questions / Help
1

Hi, community.

I have a Broadway pipeline, for some business reasons, isn’t a must this pipeline works all the time. So, some resources could be saved if the pipeline only works in a defined schedule.

I’m thinking of something like a feature flag with FunWithFlags but I’m unsure how to get this.

Other approach could be, instead of adding [MyBroadway, []] to the supervisor in application.ex, use schedules job to execute

Broadway.start_link(MyBroadway, config)

and

Broadway.stop(MyBroadway)

In this case, if I wanted to stop/resume it on demand (for some A/B testing maybe), I would have to enter the server and run the code manually in the remote console.

Thanks in advance for any suggestions.

2

One option is to call GenStage.demand(producer, :accumulate) on each producer returned by Broadway.producer_name to stop and then call GenStage.demand(producer, :forward) when you want it to resume. :slight_smile:

3 Likes
3

Thanks for the quick response!

Sorry, I forgot to mention that in this case, we have a custom producer that makes queries to the database.

I’m not sure if GenStage.demand(producer, :accumulate) have some implication in this situation :thinking:

4

It should not. But if you have a custom producer, you can also ask it to stop polling.

1 Like
5

Testing in local GenStage.demand(producer, :accumulate) and GenStage.demand(producer, :forward) seem to work fine.

But, I’m curious about how can I ask the producer to stop polling :thinking:

6

Send it a message?