pablodavila

pablodavila

Uploading large files to external API

Hello,

I’m working on a file uploader for a third-party service. The file is supposed to be sent via a POST request with the image (and metadata) encoded as base64 in the body. I set up a retry system where if I get a non-200 response, the image is added to the queue again but many requests are getting a timeout. Specifically, NimblePool throws an error mentioning a timeout but no additional details. I attempted to increase the timeout length but it seems like I get the error immediately, which isn’t re-added to the queue because the process crashes but that’s another issue. A similar (but simpler) project with Python doesn’t seem to be having the same problems but it’s not practical because it’s orders of magnitude slower.

My guess is that it has something to do with the body size but I’m not sure if I could send it as multipart. My second guess is that it has to do something with sending plenty of request at almost the same time. Not really sure if any of these are true but regardless, I’m not sure how to continue to debug or try something different.

I’m using Finch with Tesla, happens also with only Finch. HTTPoison gives me similar errors.

Thanks :slight_smile:

First Post!

pablodavila

pablodavila

Adding further details, I have 2 GenServers handling the requests: a queue and a queue watcher. The watcher takes one request from the queue and fires it using Task.start/1.

The error message mentions NimblePool.checkout timing out. Could it be that there is no available connection in the pool to perform the request? I might be firing more requests than I should.

What’s strange is that it didn’t happen the first time I ran it so it might be dependent on the external service. In that case, how can I handle the crashes/errors so I can add the request back into the queue? I tried using “try/rescue” but that doesn’t seem to work. Never had to handle this sort of errors before.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
earth10
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New

Other popular topics Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39297 209
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New

We're in Beta

About us Mission Statement