How do I retry throttled S3 puts

frumos

Hello Elixir community!

I have an issue that S3 throttles my requests and app eventually fails.

This is a function uploading stuff and been throttled by S3

def upload_file(bucket, object_key, file_path) do
    Logger.info("Upload file: #{file_path} to: #{object_key}")

    file_path
    |> ExAws.S3.Upload.stream_file()
    |> ExAws.S3.upload(bucket, object_key,
      # 240 sec
      timeout: 240_000
    )
    |> ExAws.request!()

    object_key
  end

This is S3 response:

** (EXIT from #PID<0.1257.0>) shell process exited with reason: an exception was raised:
    ** (ExAws.Error) ExAws Request Error!

{:error, {:http_error, 503, "<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>A982186D97E156DC</RequestId><HostId>gZCe7g1Hat41iziw8NYDK8K1Wz9WoHEPpq1DrNv9bVAqDB4Mh+beKYXjE8BMqPPgjeCOnSZ9unQ=</HostId></Error>"}}

        (ex_aws 2.1.3) lib/ex_aws.ex:66: ExAws.request!/2
        (kona 0.1.0) lib/remo3/aws_s3.ex:38: Remo3.Aws.S3.upload_file/3
        (kona 0.1.0) lib/remo3/account.ex:103: Remo3.Importer.Account.finalize_account_file/3
        (elixir 1.10.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
        (elixir 1.10.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
        (stdlib 3.11) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

this is ExAws config:

config :ex_aws, :retries,
  max_attempts: 30,
  base_backoff_in_ms: 30,
  max_backoff_in_ms: 60_000

Could you suggest please how can resolve the issue by continue retrying until success? What is missed in my implementation for that?

Thank you.

PS: My S3 partition schema is in good shape and been reviewed by S3 team.

7 comments

#ex_aws_s3 #throttle

1 1988 7

2020-07-28 16:26:05 UTC

Showing Posts 1 to 7

Show Best Posts
Show All (oldest first)
Show All (newest first)

NobbZ

You need to rate limit your calls to uploaf_file/3.

You seem to be calling it beyond your allowed rate.

There are several ways to do so and do depend on how you are using the function.

Post #1

frumos

Sorry I just need to persist to retry … my rate is far less that capabilities of S3 (this has been already discussed internally according to this: Best practices design patterns: optimizing Amazon S3 performance - Amazon Simple Storage Service)

The reason S3 throttles is that it detect it as anomaly burst with internal mechanism. So my goal is to get retry logic in shape. Thank you.

Post #2

NobbZ

If it’s detected as an abnormal burst, you need to limit the rate with which you try to upload.

Just retrying more often will not solve the issue. It will just make it worse, as it even adds more and more requests.

Post #3

frumos

Sorry I just need to retry with exponential backoff and this strategy to my understanding comes out of the box here: ExAws — ExAws v2.7.0 but i am not sure if it even works. At least I see no any actions in the log for that.

Post #4

kip

ex_cldr Core Team

@NobbZ is suggesting that any back off strategy is not going to produce optimism throughput and may in fact contribute to you getting rate limited messages because you will likely make the “burst” behaviour more prevalent as detected by AWS.

Using a rate limiter will maximise request throughput, create a much more event workload for AWS and therefore reduce the change you get rate limited (according to whatever AWS decides your rate should be).

Using a rate limiter with Elixir (and Phoenix if required) is quite easy. This article might be a good place to start.

Post #5