Broadway SQS delivery time

Questions / Help
1

Hello, I am not sure the problem is in the library, but maybe you could help me in what way should i research.
We use broadway_sqs library for amazon sqs.
we measure time to deliver for each message separately on the application side. Time to deliver may be up to 20 seconds!
So the question is, what may it take so long?
[{“delivered”:0.0},{“delivered”:6.0},{“delivered”:10.0},{“delivered”:10.0},{“delivered”:9.0},{“delivered”:9.0},{“delivered”:10.0},{“delivered”:9.0},{“delivered”:9.0},{“delivered”:10.0},{“delivered”:10.0},{“delivered”:9.0},{“delivered”:11.0},{“delivered”:10.0},{“delivered”:9.0},{“delivered”:11.0}]

2

Can you elaborate exactly how you’re measuring delivered?

Also keep in mind that by default broadway_sqs is configured to process messages in batches, up to some time limit. If your volume of messages is very low, you may be waiting for the time limit to hit.

3

measuring is very simple:

  • track time before we put in queue via ex_aws library
    {:ok, %{body: %{message_id: message_id}}} = ExAws.SQS.send_message(queue_url, message) |> ExAws.request()
  • track time inside method process_data
    def handle_message(_, message, _) do
    message
    |> Message.update_data(&process_data(&1, message.metadata.message_id))
    end
4

Can you show the code you have that sets up broadway?

5 
Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {
          BroadwaySQS.Producer,
          queue_url: Keyword.get(config, :queue_url), config: Keyword.get(config, :credentials)
        },
        stages: 10
      ],
      processors: [
        default: [stages: 100]
      ],
      batchers: [
        default: [
          batch_size: 10,
          batch_timeout: 2000
        ]
      ]
    )
6

Right, so at a minimum if you aren’t sending more than 10 messages you’ll have a wait time of at least 2 seconds, since that’s the batch timeout. The other delays you mentioned of up to 20 seconds sound a lot like the max value for the SQS wait_time_seconds option. You aren’t setting it explicitly, but the queue itself will also have a default. What is the queues ReceiveMessageWaitTimeSeconds value?

1 Like
7

ReceiveMessageWaitTimeSeconds = 0

If there are batchers, the acknowledgement is done by the batchers, using the batch_size

I thought batch_size needs only for acknowledgement, to collect all messages in a batch

8

I don’t understand your question sorry.

9

Even if you are not interested in working with Broadway batches via the
handle_batch/3 callback, we recommend all Broadway pipelines with SQS
producers to define a default batcher with batch_size set to 10, so
messages can be acknowledged in batches, which improves the performance
and reduce the costs of integrating with SQS.

Sorry for my english, batch_size and batch_timeout do not block producer to receive more messages. These params used only for acknowledgment (request to sqs in the end of pipeline)