Uploading large files to external API

Questions / Help
, , ,
1

Hello,

I’m working on a file uploader for a third-party service. The file is supposed to be sent via a POST request with the image (and metadata) encoded as base64 in the body. I set up a retry system where if I get a non-200 response, the image is added to the queue again but many requests are getting a timeout. Specifically, NimblePool throws an error mentioning a timeout but no additional details. I attempted to increase the timeout length but it seems like I get the error immediately, which isn’t re-added to the queue because the process crashes but that’s another issue. A similar (but simpler) project with Python doesn’t seem to be having the same problems but it’s not practical because it’s orders of magnitude slower.

My guess is that it has something to do with the body size but I’m not sure if I could send it as multipart. My second guess is that it has to do something with sending plenty of request at almost the same time. Not really sure if any of these are true but regardless, I’m not sure how to continue to debug or try something different.

I’m using Finch with Tesla, happens also with only Finch. HTTPoison gives me similar errors.

Thanks :slight_smile:

2

Adding further details, I have 2 GenServers handling the requests: a queue and a queue watcher. The watcher takes one request from the queue and fires it using Task.start/1.

The error message mentions NimblePool.checkout timing out. Could it be that there is no available connection in the pool to perform the request? I might be firing more requests than I should.

What’s strange is that it didn’t happen the first time I ran it so it might be dependent on the external service. In that case, how can I handle the crashes/errors so I can add the request back into the queue? I tried using “try/rescue” but that doesn’t seem to work. Never had to handle this sort of errors before.