std_alloc memory leak

Hello. Does anybody have any ideas how to debug standard allocator memory leak. Standard allocator is used by a lot of different types of data, and I have no ideas how to find the real problem.

Now out production service leak for about 200MB every day.

> recon_alloc:memory(allocated_types).
 {std_alloc,1_883_799_552}, %1.8GB

Check if you are not leaking processes or atoms. As far as I know, the process registry for named processes goes to that table, and if you are leaking atoms on defining named processes at runtime that might be the problem.

Thanks, but it is not that case

> length(erlang:registered()).

> erlang:memory(atom).

Now I’m going to enable intrumentation ( for std_alloc with +MDatags (it is not enabled by default) and see

Does any one process or group of processes show high memory usage?

No, memory used by system, not processes

1> erlang:memory().
 {processes,109_996_352}, %100MB there
 {system,2_209_423_256}, %2GB there

Have you read the online/free Erlang in Anger book? It is about debugging BEAM systems and has a section about debugging this specific kind of issue as I recall. :slight_smile:

9. std_alloc: catch-all allocator for whatever didn't fit the previous categories. The 
process registry for named process is there. 

By default, there will be one instance of each allocator per scheduler (and you should 
have one scheduler per core), plus one instance to be used by linked-in drivers using async 
threads. This ends up giving you a structure a bit like in Figure 7.1, but split it in N parts 
at each leaf. 

Each of these sub-allocators will request memory from mseg_alloc and sys_alloc 
depending on the use case, and in two possible ways. The first way is to act as a multiblock 
carrier (mbcs), which will fetch chunks of memory that will be used for many Erlang terms 
at once. For each mbc, the VM will set aside a given amount of memory (about 8MB 
by default in our case, which can be configured by tweaking VM options), and each term 
allocated will be free to go look into the many multiblock carriers to find some decent space 
in which to reside. 

Whenever the item to be allocated is greater than the single block carrier threshold 
(sbct) 15 , the allocator switches this allocation into a single block carrier (sbcs). A single 
block carrier will request memory directly from mseg_alloc for the first mmsbc 16 entries, 
and then switch over to sys_alloc and store the term there until it's deallocated. 

So looking at something such as the binary allocator, we may end up with something 
similar to Figure 7.2 


See whole book for details.

There is no information about leak capabilities for std_alloc

After enabling memory instrumentations for std_alloc I’ve found this

> instrument:allocations(#{allocator_types => [std_alloc]}).

re_heap =>

Our service recompile thousand regexp from time to time and store in into ets, but each time it deletes old regexps, so no memory leak should be. Will try to research further