In my project I spawn separate process for each download with HTTPotion, which works perfect on small download numbers (<5), but if I try to push higher load (~100 files) performance degrades substantially - from 20s per file to 900s per file and I start to get :timeout
errors. I tried to tune performance via :ibrowse
options, but didn’t get much improvement.
I have put together a project that demonstrates the issue: https://github.com/andre-dubber/download_issue
Is there a way keep high throughput without distributing load over multiple Erlang nodes?
P.S. I did create a separate branch in my project where I don’t use spawn
as recommended by HTTPotion author, but behaviour is still same.
Any suggestions?