Fl4m3Ph03n1x
Why is hackney consuming so much resources?
Background
I have an app that makes 2000 requests per second. To make such requests I use HTTPoison which uses hackney. Now each of this requests is fired from a specific pool with 10_000 max connections, so in theory it should be fine.
Problem
The problem is that all 8 CPUs are on fire, and hackney’s mailbox increases steadily overtime - this is translated into a steady RAM usage increase, until there is no more RAM available and bad things start to happen.
Research
To find out what is happening, I used observer and realized that the function with most reductions is hackney_pool:init followed by some others:
Also, the memory graph which indicated the mailbox is rising:
Questions
I am confused. I am not familiar with hackney_pool:init but I assume it initializes a pool (judging from the name) - which is even worst. I only create the pools once at the start, so why is this function being called all the time?
Also, is there a reason on why this is happening? Does it mean that my pool connections need to be bigger?
Marked As Solved
alco
Each row in Observer’s Processes table represents a single process. The second column displays process names for named processes (e.g. code_server, hackney_manager) or the initial func called when the process started if the process has no name (e.g. hackney_pool:init/1, tls_connection:init/1). If you look at the Current Function column, you’ll see that most processes are different OTP behaviours: gen_server, gen_statem, etc.
Back to your question, hackney’s pool implementation has known faults. See Hackney pool overload · Issue #510 · benoitc/hackney · GitHub, Pool queue usage optimization · Issue #549 · benoitc/hackney · GitHub. It doesn’t look like hackney’s maintainer has enough interest or time to get if fixed properly, so I would advice on disabling hackney’s pool and setting up your own pool of processes where each process performs a single HTTP request.
Also Liked
Manzanit0
LOL, we’re in 2020. not 2019. I don’t know why I thought this was just this month… Nevermind me 
rvirding
Does the memory usage keep growing? So for example in the next minute would it have reason above 900 Mb? Is the number of processes growing? This you can see on the first System page.
Fl4m3Ph03n1x
Yes, the memory keeps growing. However the number of processes seems to be stable at arround 24% - 26%. What does it mean?
The MsgQ for hackney_pool:init does seem to grow without bound however.
Mmmmm, would you recommend any articles on doing such? I am usually weary of re-inventing the wheel all on on my own :S
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"