tanweerdev
For no obvious reasons, my request hangs when I try to upload hundreds of images/files to aws
When request hangs control gained back is not printed(when I try to upload files bigger than 20 mb) But I see below error in my console
[warn] ExAws: HTTP ERROR: :closed for URL: "https://logs.us-east-2.amazonaws.com/" ATTEMPT: 1
I use below code to upload files
res = files_list
|> Task.async_stream(&upload_file(&1, images_upload), max_concurrency: 4, timeout: 1_500_000)
|> Stream.run()
IO.inspect "control gained back"
res
and
def upload_file({file_type, local_file_path, file_name}, %ImagesUpload{} = images_upload) do
file = File.read!(local_file_path)
destination_path = Path.join([images_upload.upload_id, @file_types[file_type], file_name])
S3.put_object(component_bucket(), destination_path, file)
|> ExAws.request!()
Logger.info("uploaded image #{destination_path}")
{:ok, destination_path}
end
and in postman I get msg
Could not get response
Error: socket hang up
Most Liked
Eiji
@tanweerdev Connection could be closed for various reasons more or less related with client. However if you did not reached any edge-case I think that you hit some timeout. Did you tried changing hackney options? Did you tried retries configuration?
I guess you have just started using this library or some configuration you have is still default (ATTEMPT: 1 so most probably no retries configuration). For bigger files and especially for slow internet connection chance for timeout may be really big, so it’s important that proper download/upload strategy is used regardless of service/software you use as everything have limits.
Also is your internet stable? I don’t think it’s that, but it’s just a good example … Some time ago fiber network I’m using was overloaded (people stay at home - read: “facebook” - because of covid restrictions) and I had many connection problems that time.
Last Post!
Eiji
ok, so most probably it’s not about content size except there is some limit on S3, but I don’t think it’s that as I believe that it should return a proper error response …
I have one more idea … Can you please compare result of:
S3.put_object(component_bucket(), destination_path, file)
with this documentation:
If it’s not that then I don’t think it’s possible to debug it properly at least from your app … Maybe we would need to debug some values in ex_aws code to have more information about request …
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security










