What have you actually tried to find the problematic processes? Have you tried listing all processes, checking their memory?

Could be related to binaries not garbage collected. Sounds like the same problem as here: https://stackoverflow.com/questions/43613027/solving-large-binaries-leak/43685158

It’s not binary if observer shows it as process memory.

Been playing with this from wobserver: logflare/lib/logflare/system_metrics/wobserver/processes.ex at master · Logflare/logflare · GitHub

Check per process memory usage. Also mailbox length.

If it’s not obvious that’s is one or a small group of processes then track process count over time. Maybe it’s a lot of smaller procs building up.

Is there a crash dump being produced when the system crashes? If so, you might find some clues if you load the dump with the crash dump viewer Crashdump Viewer — OTP 29.0.2 (observer 2.19) and if not you should try enabling them.

I was able to pin point a bottleneck on my application just last week using this technique, when I noticed I had a task supervisor with many messages in it’s mailbox at the time of the crash.

We know which processes are problematic. They belong to a group we call Workers. We identified this by checking their memory usage going up using observer_cli. It is evident that their memory grows continuously up to 30MB each, and because we have thousands of Worker processes, the machine eventually runs out of memory and crashes.

Per process memory usage keeps rising without end, but the mailboxes are empty.

It’s not process count. We have eliminated that possibility as the number of processes in the system is stable.

How do I enable a crashdump if the machine crashes while out of memory?
This technique could prove useful in deed !

The application can still produce the crash dump if it OOMs, in fact, the first couple of crash dump “slogans” documented in the link I shared earlier are exactly for that situation.
Unless you have disabled crash dumps or your server does not have a writable writable file system mounted, they should get written to the current working directory, for example I found the erl_crash.dump file sitting at the top of my release directory, right next to the bin dir. Maybe you already have one on one of your servers too. There are some env vars you can use to configure your crash dumps, documented here:

And what do they do?

These workers hold a State with a connection pid (which is in reality the PID from a gun process) and a map with streams (when making requests with gun, each request is a Stream )

Following is the state of each worker:

defmodule State do
    @moduledoc """
    State of the Worker. Most importantly, it saves a map of stream refs to
    URLs, so we know which URLs were affected when a stream breaks.
    """
    defstruct conn_pid: nil,
      active_streams: %{}
end

Now, what could possibly go wrong?

The only thing that goes through my mind is that the active_streams map may grow forever if a stream never receives the :fin_packet necessary to tell our worker the request is over (HTTP Protocol details).

But we have no reason to believe such is happening…

Is there a way to know how much memory (in bytes) a Map (or any variable) is occupying ?

You have no reason to believe the network can fail?

Touché !
That is a very good point we are overlooking. Although, IIRC, gun requests also time out. But am I not sure how the workers behave upon timeouts.

Nice catch !