pablodavila

pablodavila

Uploading large files to external API

Hello,

I’m working on a file uploader for a third-party service. The file is supposed to be sent via a POST request with the image (and metadata) encoded as base64 in the body. I set up a retry system where if I get a non-200 response, the image is added to the queue again but many requests are getting a timeout. Specifically, NimblePool throws an error mentioning a timeout but no additional details. I attempted to increase the timeout length but it seems like I get the error immediately, which isn’t re-added to the queue because the process crashes but that’s another issue. A similar (but simpler) project with Python doesn’t seem to be having the same problems but it’s not practical because it’s orders of magnitude slower.

My guess is that it has something to do with the body size but I’m not sure if I could send it as multipart. My second guess is that it has to do something with sending plenty of request at almost the same time. Not really sure if any of these are true but regardless, I’m not sure how to continue to debug or try something different.

I’m using Finch with Tesla, happens also with only Finch. HTTPoison gives me similar errors.

Thanks :slight_smile:

First Post!

pablodavila

pablodavila

Adding further details, I have 2 GenServers handling the requests: a queue and a queue watcher. The watcher takes one request from the queue and fires it using Task.start/1.

The error message mentions NimblePool.checkout timing out. Could it be that there is no available connection in the pool to perform the request? I might be firing more requests than I should.

What’s strange is that it didn’t happen the first time I ran it so it might be dependent on the external service. In that case, how can I handle the crashes/errors so I can add the request back into the queue? I tried using “try/rescue” but that doesn’t seem to work. Never had to handle this sort of errors before.

Where Next?

Popular in Questions Top

jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

We're in Beta

About us Mission Statement