pdilyard

Extremely high memory usage in GenServers

I’m deep into debugging a very high memory usage problem in a group of GenServers.

There are two types of GenServer implementations I’m examining:

# module:
MessageEngine.Thought

# example state:
%DB.Thought{__meta__: #Ecto.Schema.Metadata<:loaded, "thoughts">, active: false,
 score: 0.35795454545454547,
 conversation: #Ecto.Association.NotLoaded<association :conversation is not loaded>,
 conversation_id: 1621, id: 129158,
 inserted_at: #Ecto.DateTime<2017-03-07 21:32:19>,
 lost_against: %{"129129" => [51952, 51955, 51955, 51938, 51931, 51951, 51944], ...},
 message: #Ecto.Association.NotLoaded<association :message is not loaded>,
 message_id: 12748,
 text: "Yes because will she listen to them or the people.",
 updated_at: #Ecto.DateTime<2017-03-07 21:44:21>,
 user: #Ecto.Association.NotLoaded<association :user is not loaded>,
 user_id: 51959, vector: [],
 won_against: %{"129129" => [51946, 51934, 51934, 51942, 51954, 51957], ...}}

# module:
MessageEngine.User

# example state:
%DB.MessageUser{__meta__: #Ecto.Schema.Metadata<:loaded, "messages_users">,
 accepting_choices: false,
 all_choices: [%{"c" => 129138, "nc" => 129154}, ...],
 comparisons: [%{"a" => 129138, "b" => 129154},  ...],
 conversation: #Ecto.Association.NotLoaded<association :conversation is not loaded>,
 conversation_id: 1621, id: 132055,
 inferred_choices: [%{"c" => 129138, "nc" => 129154}, ...],
 manual_choices: [%{"c" => 129138, "nc" => 129130}, ...],
 message: #Ecto.Association.NotLoaded<association :message is not loaded>,
 message_id: 12748, rid: nil,
 user: #Ecto.Association.NotLoaded<association :user is not loaded>,
 user_id: 51959}

I don’t want to dig too deeply into why the states are what they are, but suffice to say that they have been well-researched and tested, and I don’t want to explain too much industry context

Now, we have been monitoring our app in production for a while, and noticed that, as the number of these processes alive increase, memory usage goes up almost exponentially.

With 600 MessageEngine.Users and 600 MessageEngine.Thoughts, we measured almost 35GB of RAM being used across the cluster.

I first tried to measure the amount of memory used just by the state of the process, but this doesn’t seem like nearly enough data to have that substantial of an impact.

I popped into observer to learn more, and ran the following tests:

30 users and 30 thoughts

With:
length(MessageEngine.User.all_choices) = 0
length(MessageEngine.User.manual_choices) = 0
length(MessageEngine.User.inferred_choices) = 0
length(MessageEngine.User.comparisons) = 0

One MessageEngine.User process was consuming 139kb of memory
One MessageEngine.Thought process was consuming 3kb of memory

With:
length(MessageEngine.User.all_choices) = 53
length(MessageEngine.User.manual_choices) = 20
length(MessageEngine.User.inferred_choices) = 33
length(MessageEngine.User.comparisons) = 53

One MessageEngine.User process was consuming 502kb of memory
One MessageEngine.Thought process was consuming 25kb of memory

300 users and 300 thoughts

With:
length(MessageEngine.User.all_choices) = 0
length(MessageEngine.User.manual_choices) = 0
length(MessageEngine.User.inferred_choices) = 0
length(MessageEngine.User.comparisons) = 0

One MessageEngine.User process was consuming 1089kb of memory
One MessageEngine.Thought process was consuming 6kb of memory

With:
length(MessageEngine.User.all_choices) = 53
length(MessageEngine.User.manual_choices) = 20
length(MessageEngine.User.inferred_choices) = 33
length(MessageEngine.User.comparisons) = 53

One MessageEngine.User process was consuming 4023kb of memory
One MessageEngine.Thought process was consuming 41kb of memory

So, as you can see, not only is memory usage per-process scaling up a lot just by adding ~50 maps to a list, the usage of each process also seems to be dependent on the number of processes alive! An order of magnitude increase in the number of processes results in an order of magnitude increase in the memory usage of each one.

This seems like really weird behavior to me, and I’m kinda stuck on where to go next, because, by my calculations, the memory usage of the state of these processes should be more like 20-50kb each (used this guide: Memory Usage — Erlang System Documentation v29.0.2).

Here’s a full dump of the state of a process that was using 4023kb of RAM: Memory usage · GitHub

Any help would be greatly appreciated.

33 comments

#otp #troubleshooting #genserver

130 14072 33

2019-11-06 00:30:57 UTC

Marked As Solved

sasajuric

Author of Elixir In Action

It looks like you have one or more processes that are touching a “large” binary (i.e. a binary > 64 bytes), but are not allocating data frequently enough to be garbage collected themselves.

A large binary is reference counted, instead of being copied across processes. A reference count is bumped by every process that touches such binary. When a reference goes out of scope, the count is going to be decremented only after a fullsweep GC takes place. Until then, the ref count of a binary is > 0, and it’s kept in memory even if no one uses it. Therefore, if you have at least one process that touched a binary in the past, but is not allocating data too frequently to trigger a “fullsweep” GC, you’ll end up with an excessive amount of garbage binaries.

A simple example could be a process that acts as a mediator. It receives a message, then dispatches it to another process, and does nothing more than that. It doesn’t allocate a lot of data on its own, so it’s going to be GCed less frequently. If a part of dispatched messages is a large binary, the process touches a lot of large binaries, and can therefore be the cause of excessive dangling garbage.

You first need to identify such processes. Judging by your other output, it looks like they could be your User processes, but I can’t say for sure.

Once you know which processes are causing the problem, a simple fix could be to hibernate the process after every message. This is done by including :hibernate in the result tuple of handle_* callbacks (e.g. {:noreply, next_state, :hibernate}). This will reduce the throughput of the process, but can do wonders for your memory usage.

Another option is to set the fullsweep_after flag of the problematic process to zero or a very small value. I think that GenServer.start_link(callback_module, spawn_opt: [fullsweep_after: desired_value]) should do the job. For more explanation, look for fullsweep_after in docs for the :erlang module.

Post #23

Also Liked

dominicletz

Creator of Elixir Desktop

Bit late but another useful new feature to start your gen_server with the hibernate_after option, such as:

{:ok, worker} = GenServer.start_link(module, args, hibernate_after: 5_000)

This will ensure that once your worker is bored for more than 5 seconds it will garbage collect everything it can.

Post #33

michalmuskala

I haven’t really closely followed the discussion, so this might be a bit misplaced. But a common pattern for handling memory-expensive operations inside a GenServer is to spawn a separate process to do the processing - this means the process itself does not grow extensively in size, and the memory used for the computation can be freed immediately (when the “operation” process terminates) - you could even consider starting the process with a bigger initial heap to eliminate GC completely (though, that might be risky and excessive without thorough measurement).

For example, this could look like this:

def handle_call(_req, from, state) do
  task = Task.async(fn ->
    # some computation
  end)
  {:reply, Task.await(task), state}
end

Or in case the response could be delivered asynchronously, even like this:

def handle_call(_req, from, state) do
  Task.start_link(fn ->
    # some computation
    GenServer.reply(from, reply)
  end)
  {:noreply, state}
end

Post #27

dom

bin_leaks forces a garbage collect on all processes, and measures how many reference-counted binaries were freed per process. So this confirms lack of GC is the issue here.

Some things you can do:

If you have operations that generate lots of refc binary garbage, do them in a separate, short-lived process linked to your long-lived user process, so it doesn’t accumulate garbage.
You can use a timer to hibernate (see genserver doc) the user process after N seconds of inactivity, or when you know it won’t be getting messages for a while. The process will still be alive, but won’t hold extra memory.
You can also use a timer to force a gc every N seconds.
ETS as mentioned can help. Each process can own a table, it doesn’t have to be shared. This is a nice article about the difference it makes: The Erlangelist - Reducing the maximum latency

Post #18