Why does garbage collection not work as intended?

If I understood correctly what is happening is that large binaries are not stored in the heap. so the process doesn’t get their heap full and don’t trigger the GC… what ends up keeping the large binaries allocated for a longer perioed of time, by the beam book:

This means that binaries, which has a tendency to be large or even huge, can hang around for a long time after all references to the binary are dead. Note that since binaries are allocated globally, all references from all processes need to be dead, that is all processes that has seen a binary need to do a GC.
The BEAM Book: Understanding the Erlang Runtime System

Thanks! That sounds like a logical suggestion, but I don’t know how to explain the fact that memory usage dropped at some point. I thought that could only happen after a garbage collection run. And then, if there was a garbage collection run, I don’t get why so much memory (15MB) was not reclaimed.

I don’t think this is the binary “leak” per se; because you are looking at the process memory, not the global binary heap. It is certainly related. The core issue is still GC not triggered frequent enough. Infrequent memory intensive operation in a long live process may not be the best use of available memory. You can:

• hibernate like you did, or
• spawn or use a short lived task for the memory intensive operation

it’s difficult to answer without knowning exactly what happens in that genserver, but I’m assuming that the genserver is doing whatever he is supposed to do. at the moment it triggers the GC it gonna keep only “recent data”, so it keeps the reference to a few versions of this large binary. since all the stuff that you’re doing are out of heap(large binary, ets table) it very rarely triggers the GC.

as per solution, you can do as @derek-zhou suggested.

Thanks again. I was assuming that the LiveDashboard view of the process would somehow include the binaries, but now I think of it that probably does not make much sense.

I might have time later to prepare a minimal reproducible example in code, but in the meantime, I would be very interested to know how to investigate the cause of this in the first place (the solution is clear, and in fact I am using hibernation right now). Since Erlang has been out there for so long, I assume there must be tooling to diagnose garbage collection issues. Or is looking at the code the only way to do it? I can imagine I will come across this kind of problems in the future.

I guess you are probably already aware of it, but if not, you can use :observer.start() to spawn Erlang’s observer and there you can see how much binary is being used globally, per process, per ets table, etc.

is this biting you in anyway? or you’re just striving for very low memory use?
unless you’re running on embedded devices, i woudn’t worry about that… if that’s the case, you can always change the max_heap_size for the entire erlang node or for individual processes. and you can trigger garbage collection manually on a process basis too.

but again, i would only bother with that if that’s hitting you in some way, otherwise it doesn’t pays the effort of tunning things just for the sake of low memory consumption.

If you do a trace on the process for garbage collection events you will get this information whenever a GC is run: erlang — OTP 29.0.2 (erts 17.0.2)

To start such a trace you can use the runtime_tools module :dbg:

:dbg.tracer(), :dbg.p(pid, [:garbage_collection])

You can also get some information from a running process by calling :erlang.process_info(pid, :garbage_collection_info).

Are you parsing parts of the binary data and lifting binaries derived from it into the ets table? This will result in the ets table “holding on” to the binary data because each of those smaller binaries is kept as a reference to the parent one, which cannot be garbage collected. Then finally when you flush the ets table and replace it, the parent process gc’s the ets reference and that in turn allows the gc’s to finally let go of the initial"huge binary".

In general when stashing content into an Ets table it’s probably a good idea to copy the binary. If you’re parsing huge jsons, use the copy binary option instead of the reference binary (default) option.

You should probably also consider directly deleting the ets table when you refersh instead of relying on the genserver to GC the reference which may or may not happen when you expect.

Thank you all for the great answers! My takeaways (also after reading other posts) are:

1. The process memory displayed by LiveDashboard does not account for reference counted binaries (except for the size of the ProcBins).
2. Garbage collection can take place before the end of a function, which increases the chance of stuff not being cleaned up properly, because objects that will later be thrown away are still being used.
3. As the heap size of a process grows, garbage collection will be triggered less frequently.
4. Using :hibernate is not as exceptional as it sounds (e.g. LiveView uses hibernate_after with a default of 15 seconds).