adobroskok
In my project I spawn separate process for each download with HTTPotion, which works perfect on small download numbers (<5), but if I try to push higher load (~100 files) performance degrades substantially - from 20s per file to 900s per file and I start to get :timeout errors. I tried to tune performance via :ibrowse options, but didn’t get much improvement.
I have put together a project that demonstrates the issue: GitHub - andre-dubber/download_issue · GitHub
Is there a way keep high throughput without distributing load over multiple Erlang nodes?
P.S. I did create a separate branch in my project where I don’t use spawn as recommended by HTTPotion author, but behaviour is still same.
Any suggestions?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
dimitarvp
I know this is probably not what you are looking for but, have you tried using
:httpcorHTTPoison(which uses:hackney)?Alternatively, better use Tesla and switch underlying HTTP client for easier testing and evaluation.
adobroskok
Based on this article: http://big-elephants.com/2016-12/http-streaming-in-elixir/ I have decided to go in favour of HTTPotion/:ibrowse as in my business scenario there is a possibility of thousands concurrent downloads and I need to avoid memory usage spikes.