Broadway Message Distribution Strategy

Questions / Help
, ,
3

Here is the broadway setup I tried in the above screenshot. Previously I tried 200 workers with the same effect. I can try 20k but I definitely don’t want to be processing that many files at a time. The files size range unfortunately from a few kb to 200-300 MB so their weights are probably not being respected.

    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producers: [
        default: [
          module:
            {BroadwaySQS.Producer,
             queue_name: "my-pipeline-prototype",
             max_number_of_messages: 10,
             wait_time_seconds: 20,
             visibility_timeout: 300,
             receive_interval: 10},
          stages: 50
        ]
      ],
      processors: [
        # downloaders: [
        #   max_demand: 1,
        #   stages: System.schedulers_online()
        # ],
        default: [
          max_demand: 1,
          # + div(System.schedulers_online(), 2)
          stages: 2000
        ]
      ],
      batchers: [
        default: [
          batch_size: 10,
          batch_timeout: 10_000,
          stages: 50
        ]
      ]
    )

There is more information in my last post:

1 Like