:erts_debug.size/1
Maybe there are even old copies of values lingering around that don’t get GC’d because never a GC is triggered?
Have you tried to occasionally hibernate those processes to enforce GC? This might have a negative impact on its latency though.
Each worker calls :erlang.garbage_collect() every 15 seconds, though this doesn’t seem to be doing much. How is that hibernating a process would help me? Could you elaborate?
If a manual GC helps they’re likely not large, they’ve just been updated a bunch.
So, your best options are hibernate, or move that state to ETS.
Not sure but that seems excessive and in general bad practice unless you really know what you’re doing (which I don’t so I avoid it 
).
But if you inspect the state of one of those 30MB processes you can at least see how big that map is and if it’s really 30MB of streams.
If it’s not really 30MB of connections then I think just moving that state to ETS will fix it. It should be a pretty quick change so a low risk test vs futsing around with a bunch of GC settings/calls.
We really don’t know what we are doing. We’re just desperately trying to fix this memory issue. Anything goes at this point 
How so? If my connections map is small, (say 700K while the processes memory occupies 30MB), how is is that moving the map to an ETS table will help? Wouldn’t that just make the ETS memory usage rise with no end instead of the process memory?
Can you elaborate?
First, I’ll say that garbage collection with the VM is a bit of a black box for me. If I suspect a GC issue I just try and eliminate that possibility altogether so that, if it’s not a GC issue, I can move on. You do this by not keeping data in a long running process’ state. You move it to ETS.
Process state is garbage collected. ETS is not. When you update data in ETS there are no extra copies in memory. When you update state in the process, copies of that state build up on the heap. Garbage collections runs whenever it runs. If you have a lot of long running processes this can lead to memory issues.
For example, lets say you’re updating state once a minute. That won’t build up that fast, but then something happens to cause you to add to that map 10 times a second. This happens for 10 minutes. So the heap builds up (lots of copies of a map getting larger), but then the activity there goes back to updating state once a minute. So now we have a large heap that won’t get garbage collected soon because that state isn’t getting updating frequently enough anymore.
This specific example I’ve dealt with because I keep queues in state in a lot of long running processes. They stay fairly empty as long as my workers are working, but I’ve tested this scenario if they were to get backed up and this is exactly what happens. Memory bloats exponentially faster because not only do I have data in my queues but I have every iteration of that queue on the heap.
I use :observer_cli.start to peek into those states. Nothing unusual for a a state that has a map of connections, at least the way I see it.