PurgePJ
I have posted this in StackOverflow as well
I have been into a little trouble lately: The memory used by GenServer processes is super high, probably because of large binary leaks.
The problem comes from here: we receive large binaries through the GenServer and we pass them to the consumer, which then interacts with that data. Now, these large binaries are never assigned to a variable and the GC doesn’t go over them.
I have tried hibernating the processes after managing the data, which partially worked because the memory used by processes lowered a lot, but since binaries were not getting GC’d, the amount of memory used by them increased slowly but steadily, from 30 MBs without hibernating to 200MBs with process hibernation in about 25 minutes.
I have also tried to set :erlang.system_flag(:fullsweep_after, 0), which has also worked and lowered the memory used by processes by around 20%.
Before and after.
I must say it goes down to 60-70MB used by processes from time to time.
Edit: Using :recon.bin_leak(15) frees a lot of memory – result of :recon.bin_leak(15)
Anyhow the memory used is still high and I’m completely sure it can be fixed.
Here you have a screenshot taken from the observer in the Processes tab. As you can see, GenServer is the one eating the memory like the cookie monster.
I have researched a lot about this topic, tried all the suggestions and possible solutions that were given out there, and nevertheless, I am still in this position.
Any help is welcome.
The code is in this Github Repository
Code of interest that is probably causing this + Applications tree. 3 out of 4 processes there (<0.294.0>, <0.295.0>, <0.297.0> are using 27MB of memory.
Thank you beforehand for reading.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 10 of 18 Posts
xlphs
I work on a server that processes a lot of video and audio in real time as well, one of the first things I did is to write my own data pipeline workflow logic, for that I used only OTP. I see you are using gen_stage, after a quick glance I noticed this library buffers data and I will quote the docs:
From what you described, it feels like you are not consuming/processing data fast enough and the buffer for producer_consumer keeps growing.
I would stay away from gen_stage, or at least make a very simple version of your server that does not use it, just for the sake of comparison.
sribe
Have you tried invoking GC from producer as well as consumers?
PurgePJ
Invoking manually the GC in an interval would not be a solution to the problem but a temporary fix for the memory usage. The solution would be fixing the root cause of the problem.
PurgePJ
I will take that into account, would it be a huge change removing gen_server?
So as you say, OTP would propably process data faster or?
DianaOlympos
What @xlphs said is that you should do a test with a simple GenServer instead of trying to use GenStage. GenStage try to buffer data to ensure that all stage have always something to do, and in this case, it means the data stays in the buffer. If you stop buffering then it will not stays and the GC will be able to clean it properly.
benwilson512
You may be able to simulate this by simply changing the demand values to 1 to avoid buffering entirely. If this causes the desired change then the memory usage isn’t due to a leak, it’s just due to a buffer configured for more than you want.
PurgePJ
Do you mean setting the option of demand_value to 1 for gen stage?
benwilson512
This API has changed a little bit since I first used GenStage, but yeah you’d lower your max_demand to 1 on the consumer and producer consumer. The reason is that if you stick with the default it’ll buffer up to MIN_DEMAND which is 5_000 items before actually pushing it to the consumers.
This option will look like:
PurgePJ
Very interesting, will really have a look at it. I will later post here the results
dimitarvp
Did you end up solving your problem? Quite an interesting thread.