@tanweerdev Connection could be closed for various reasons more or less related with client. However if you did not reached any edge-case I think that you hit some timeout. Did you tried changing hackney options? Did you tried retries configuration?
I guess you have just started using this library or some configuration you have is still default (ATTEMPT: 1 so most probably no retries configuration). For bigger files and especially for slow internet connection chance for timeout may be really big, so it’s important that proper download/upload strategy is used regardless of service/software you use as everything have limits.
Also is your internet stable? I don’t think it’s that, but it’s just a good example … Some time ago fiber network I’m using was overloaded (people stay at home - read: “facebook” - because of covid restrictions) and I had many connection problems that time.
What do you think about playing a bit with multipart_chunksize? Maybe it could be good for you to set to something like 10mb or maybe even smaller … For sure it’s more like workaround, but could also optimize your project. Again I did not made such things before, but it should work as ex_aws provides a way to read configuration from Amazon CLI files.
We upload 50 mb zipped file for testing purpose. Potentially file can be as big as 3 GB according to requirements at least in production. We have to unzip the file, upload each individual image/document which isnt normally bigger than few mbs.
if I have inactivity_timeout: :infinity, it will process all the files but will not send back response even after processing it. I found this article helpful but even this doesnt provide details about the protocol possible options. As soon as All tasks are processed via async_stream (I have also tried to use chunk of 4 and then do task await or Task.async_stream) I need to insert some record in db and send back response. So in my opinion solution lies in http protocol options but as documentation is not very clear. I cant figure out how to achieve this. Thanks in advance for all the effort and help @Eiji