Issues with starting multiple Broadway pipelines with DynamicSupervisor

Questions / Help
,
1

Is it possible to start a Broadway pipeline with a DynamicSupervisor? Well, it is possible start at least one but when I try to start multiple only the first one starts.

I tried overriding the default child_spec with

%{
    id: "#{__MODULE__}:#{unique_id}",
    start: {__MODULE__, :start_link, [destination: destination]},
    shutdown: :infinity
}

but then none of the pipelines start and I am getting zero errors.

I overrode the default child_spec because this is the implementation:

default = %{
          id: unquote(module),
          start: {__MODULE__, :start_link, [arg]},
          shutdown: :infinity
        }

I have also defined a process_name:

  @impl Broadway
  def process_name({:via, module, {registry, name}}, base_name) do
    {:via, module, {registry, "#{base_name}:#{name}"}}
  end

This is how I am starting the pipeline:

DynamicSupervisor.start_child(
    ER.DynamicSupervisor,
    {ER.Destinations.Pipeline.S3, [destination: destination]}
)

  def start_link(opts) do
    broadway_config = get_broadway_config(opts[:destination])

    context = %{
      destination: broadway_config.destination
    }

    result =
      Broadway.start_link(__MODULE__,
        name: broadway_config.name,
        producer: [
          module:
            {OffBroadwayEcto.Producer,
             client: {ER.Destinations.Pipeline.Client, [destination: broadway_config.destination]}}
        ],
        context: context,
        processors: [
          default: [concurrency: broadway_config.processor_concurrency]
        ],
        batchers: [
          s3: [
            concurrency: broadway_config.batcher_concurrency,
            batch_size: broadway_config.batch_size,
            batch_timeout: broadway_config.batch_timeout
          ]
        ]
      )

Anyone have any ideas what I could be doing it wrong? Or if it is possible? Thanks

2

Have you been able to solve this? Admittedly I can’t offer help as I haven’t worked with Broadway for a while – but I also never tried what you do.

3

Sadly I have not yet solved the problem. I did open an issue on the Broadway repo on Github. At this point I am thinking it is something internal to Broadway that is the issue but always open to it being my code. :wink:

If I can’t structure my pipelines this way I have been thinking about alternatives. I just don’t like anything that I have come up with yet. I have thought about trying to use a regular supervisor and not a dynamic supervisor but I just don’t know why that difference would matter. I am not an expert on the differences between the two so maybe there is something there that is causing an issue.