For no obvious reasons, my request hangs when I try to upload hundreds of images/files to aws

tanweerdev

When request hangs control gained back is not printed(when I try to upload files bigger than 20 mb) But I see below error in my console

[warn] ExAws: HTTP ERROR: :closed for URL: "https://logs.us-east-2.amazonaws.com/" ATTEMPT: 1

I use below code to upload files

res =  files_list
    |> Task.async_stream(&upload_file(&1, images_upload), max_concurrency: 4, timeout: 1_500_000)
    |> Stream.run()
IO.inspect "control gained back"
res

and

def upload_file({file_type, local_file_path, file_name}, %ImagesUpload{} = images_upload) do
    file = File.read!(local_file_path)
    destination_path = Path.join([images_upload.upload_id, @file_types[file_type], file_name])

    S3.put_object(component_bucket(), destination_path, file)
    |> ExAws.request!()

    Logger.info("uploaded image #{destination_path}")
    {:ok, destination_path}
end

and in postman I get msg

Could not get response
Error: socket hang up

15 comments

#ex_aws #ex_aws_s3 #aws #stream #aws-s3 #async_stream

2 4494 15

2020-07-29 20:16:05 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

Eiji

@tanweerdev Connection could be closed for various reasons more or less related with client. However if you did not reached any edge-case I think that you hit some timeout. Did you tried changing hackney options? Did you tried retries configuration?

I guess you have just started using this library or some configuration you have is still default (ATTEMPT: 1 so most probably no retries configuration). For bigger files and especially for slow internet connection chance for timeout may be really big, so it’s important that proper download/upload strategy is used regardless of service/software you use as everything have limits.

Also is your internet stable? I don’t think it’s that, but it’s just a good example … Some time ago fiber network I’m using was overloaded (people stay at home - read: “facebook” - because of covid restrictions) and I had many connection problems that time.

Post #1

tanweerdev

I see, in some cases it retried upto 5 times. I have added provided 25 mins timeout option Task.async_stream but before even 25 mins, request is closed and control is not returned back

Post #2

tanweerdev

It could be the reason. But I and also my friend has tried on stable internet connection and same happens when files total size is more than 20 mb

Post #3

tanweerdev

The problem is I am always returning {:ok, some_response} so it should give back cotrol even if something didnt go expected

Post #4

tanweerdev

No. But Trying now

Post #5

tanweerdev

Now I have added below configs

config :ex_aws, :retries,
  max_attempts: 10

config :ex_aws, :hackney_opts,
  recv_timeout: 1_500_000,
  timeout: 1_500_000

and

config :cybord, MyApp.Endpoint,
  http: [
    port: 4000,
    protocol_options: [
      max_request_line_length: 99_999_999_999,
      idle_timeout: :infinity
    ]
  ]
...

Just to be on safe side

Post #6

Eiji

Hmm … previously I used only smaller files, so I don’t really know what may go wrong, but I found this site:

What do you think about playing a bit with multipart_chunksize? Maybe it could be good for you to set to something like 10mb or maybe even smaller … For sure it’s more like workaround, but could also optimize your project. Again I did not made such things before, but it should work as ex_aws provides a way to read configuration from Amazon CLI files.

Post #7

tanweerdev

Could it be related to socket or request processing? why I think it could be because as soon as I see below msg

Could not get response
Error: socket hang up

All the processing stops for sure. Even if there is no warning or something
Note: Last time POST request hang up after 6-7 mins although I have 25 mins timeout option almost everywhere

Post #8

Eiji

I’m of course not sure, but asked myself if there is a problem with 20mb file then how it would work in case we would send exactly same file in smaller chunks?

ok, I see that most probably I would not able to reproduce that … I have 100/100 Mbps fiber internet, so sending 20mb file would definitely not take so much time

I saw similar problems in other libraries (other languages) and there with exactly same error people found that the configuration was incorrect, but it does not looks like your use case …

Post #9

tanweerdev

We upload 50 mb zipped file for testing purpose. Potentially file can be as big as 3 GB according to requirements at least in production. We have to unzip the file, upload each individual image/document which isnt normally bigger than few mbs.

Post #10

Load more posts (5 remaining)