adobroskok
In my project I spawn separate process for each download with HTTPotion, which works perfect on small download numbers (<5), but if I try to push higher load (~100 files) performance degrades substantially - from 20s per file to 900s per file and I start to get :timeout errors. I tried to tune performance via :ibrowse options, but didn’t get much improvement.
I have put together a project that demonstrates the issue: GitHub - andre-dubber/download_issue · GitHub
Is there a way keep high throughput without distributing load over multiple Erlang nodes?
P.S. I did create a separate branch in my project where I don’t use spawn as recommended by HTTPotion author, but behaviour is still same.
Any suggestions?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 2 of 2 Posts
dimitarvp
I know this is probably not what you are looking for but, have you tried using
:httpcorHTTPoison(which uses:hackney)?Alternatively, better use Tesla and switch underlying HTTP client for easier testing and evaluation.
adobroskok
Based on this article: http://big-elephants.com/2016-12/http-streaming-in-elixir/ I have decided to go in favour of HTTPotion/:ibrowse as in my business scenario there is a possibility of thousands concurrent downloads and I need to avoid memory usage spikes.