HTTPotion/:ibrowse slows down under load

In my project I spawn separate process for each download with HTTPotion, which works perfect on small download numbers (<5), but if I try to push higher load (~100 files) performance degrades substantially - from 20s per file to 900s per file and I start to get :timeout errors. I tried to tune performance via :ibrowse options, but didn’t get much improvement.
I have put together a project that demonstrates the issue: https://github.com/andre-dubber/download_issue
Is there a way keep high throughput without distributing load over multiple Erlang nodes?

P.S. I did create a separate branch in my project where I don’t use spawn as recommended by HTTPotion author, but behaviour is still same.

Any suggestions?

1 Like

I know this is probably not what you are looking for but, have you tried using :httpc or HTTPoison (which uses :hackney)?

Alternatively, better use Tesla and switch underlying HTTP client for easier testing and evaluation.

Based on this article: http://big-elephants.com/2016-12/http-streaming-in-elixir/ I have decided to go in favour of HTTPotion/:ibrowse as in my business scenario there is a possibility of thousands concurrent downloads and I need to avoid memory usage spikes.