First, I’ll say that garbage collection with the VM is a bit of a black box for me. If I suspect a GC issue I just try and eliminate that possibility altogether so that, if it’s not a GC issue, I can move on. You do this by not keeping data in a long running process’ state. You move it to ETS.

Process state is garbage collected. ETS is not. When you update data in ETS there are no extra copies in memory. When you update state in the process, copies of that state build up on the heap. Garbage collections runs whenever it runs. If you have a lot of long running processes this can lead to memory issues.

For example, lets say you’re updating state once a minute. That won’t build up that fast, but then something happens to cause you to add to that map 10 times a second. This happens for 10 minutes. So the heap builds up (lots of copies of a map getting larger), but then the activity there goes back to updating state once a minute. So now we have a large heap that won’t get garbage collected soon because that state isn’t getting updating frequently enough anymore.

This specific example I’ve dealt with because I keep queues in state in a lot of long running processes. They stay fairly empty as long as my workers are working, but I’ve tested this scenario if they were to get backed up and this is exactly what happens. Memory bloats exponentially faster because not only do I have data in my queues but I have every iteration of that queue on the heap.

What have you actually tried to find the problematic processes? Have you tried listing all processes, checking their memory?

Yes, we figured. Turns out it was the issue described by @chasers. I was quite hard to find.