Profiling memory usage in given process

A few things occur to me as to why that may be happening:

cast them out!

Are you using cast ?
We had an issue sometime ago where we were flooding a process with messages via cast, which caused its mailbox to grow indefinitely causing it to crash the BEAM VM.

queues

Using queues?
Queues that grow without boundary (looking at you hackney) misbehave spectacularly, consuming all the system’s memory until a crash is inevitable.

https://ferd.ca/handling-overload.html

Tools you can use

If you want a specific tool, then I know of erlyberly, which shows you memory and messages passed between processes.

You can also find other tools in this discussion: