What’s the best way to run a once-off Broadway pipeline?
Background:
I am going to modify 1M postgres records published from a GenStage, but I only want to run this Broadway pipeline once as part of a prep for data migration between systems.
Is there something like a GenServer.call(Stack, :pop) I can call on the pipeline to kick it off from a custom mix task? Ideally I would start_link the pipeline to the supervisor, process all the records, then just have the mix task end and nothing is left running.
Is this possible, or is there a better way to do this? Thanks.
While I’m here, is this going to be ok to :stop like this to have Broadway “finish”, or is the supervisor tree just going to restart it?
def handle_demand(demand, state) when demand > 0 do
results =
query()
|> paginate(state.offset, demand)
|> Repo.all()
if length(results) == 0 do
{:stop, "processed all records", state}
else
{:noreply, results, %{state | offset: state.offset + demand}}
end
end
