I have a Broadway pipeline, for some business reasons, isn’t a must this pipeline works all the time. So, some resources could be saved if the pipeline only works in a defined schedule.
I’m thinking of something like a feature flag with FunWithFlags but I’m unsure how to get this.
Other approach could be, instead of adding
[MyBroadway, ] to the supervisor in
application.ex, use schedules job to execute
In this case, if I wanted to stop/resume it on demand (for some A/B testing maybe), I would have to enter the server and run the code manually in the remote console.
Thanks in advance for any suggestions.
One option is to call
GenStage.demand(producer, :accumulate) on each producer returned by
Broadway.producer_name to stop and then call
GenStage.demand(producer, :forward) when you want it to resume.
Thanks for the quick response!
Sorry, I forgot to mention that in this case, we have a custom producer that makes queries to the database.
I’m not sure if
GenStage.demand(producer, :accumulate) have some implication in this situation
It should not. But if you have a custom producer, you can also ask it to stop polling.
Testing in local
GenStage.demand(producer, :accumulate) and
GenStage.demand(producer, :forward) seem to work fine.
But, I’m curious about how can I ask the producer to stop polling