Issues with starting multiple Broadway pipelines with DynamicSupervisor

Is it possible to start a Broadway pipeline with a DynamicSupervisor? Well, it is possible start at least one but when I try to start multiple only the first one starts.

I tried overriding the default child_spec with

    id: "#{__MODULE__}:#{unique_id}",
    start: {__MODULE__, :start_link, [destination: destination]},
    shutdown: :infinity

but then none of the pipelines start and I am getting zero errors.

I overrode the default child_spec because this is the implementation:

default = %{
          id: unquote(module),
          start: {__MODULE__, :start_link, [arg]},
          shutdown: :infinity

I have also defined a process_name:

  @impl Broadway
  def process_name({:via, module, {registry, name}}, base_name) do
    {:via, module, {registry, "#{base_name}:#{name}"}}

This is how I am starting the pipeline:

    {ER.Destinations.Pipeline.S3, [destination: destination]}
  def start_link(opts) do
    broadway_config = get_broadway_config(opts[:destination])

    context = %{
      destination: broadway_config.destination

    result =
        producer: [
             client: {ER.Destinations.Pipeline.Client, [destination: broadway_config.destination]}}
        context: context,
        processors: [
          default: [concurrency: broadway_config.processor_concurrency]
        batchers: [
          s3: [
            concurrency: broadway_config.batcher_concurrency,
            batch_size: broadway_config.batch_size,
            batch_timeout: broadway_config.batch_timeout

Anyone have any ideas what I could be doing it wrong? Or if it is possible? Thanks

Have you been able to solve this? Admittedly I can’t offer help as I haven’t worked with Broadway for a while – but I also never tried what you do.

Sadly I have not yet solved the problem. I did open an issue on the Broadway repo on Github. At this point I am thinking it is something internal to Broadway that is the issue but always open to it being my code. :wink:

If I can’t structure my pipelines this way I have been thinking about alternatives. I just don’t like anything that I have come up with yet. I have thought about trying to use a regular supervisor and not a dynamic supervisor but I just don’t know why that difference would matter. I am not an expert on the differences between the two so maybe there is something there that is causing an issue.