Extremely high memory usage in GenServers

pdilyard

I’m deep into debugging a very high memory usage problem in a group of GenServers.

There are two types of GenServer implementations I’m examining:

# module:
MessageEngine.Thought

# example state:
%DB.Thought{__meta__: #Ecto.Schema.Metadata<:loaded, "thoughts">, active: false,
 score: 0.35795454545454547,
 conversation: #Ecto.Association.NotLoaded<association :conversation is not loaded>,
 conversation_id: 1621, id: 129158,
 inserted_at: #Ecto.DateTime<2017-03-07 21:32:19>,
 lost_against: %{"129129" => [51952, 51955, 51955, 51938, 51931, 51951, 51944], ...},
 message: #Ecto.Association.NotLoaded<association :message is not loaded>,
 message_id: 12748,
 text: "Yes because will she listen to them or the people.",
 updated_at: #Ecto.DateTime<2017-03-07 21:44:21>,
 user: #Ecto.Association.NotLoaded<association :user is not loaded>,
 user_id: 51959, vector: [],
 won_against: %{"129129" => [51946, 51934, 51934, 51942, 51954, 51957], ...}}

# module:
MessageEngine.User

# example state:
%DB.MessageUser{__meta__: #Ecto.Schema.Metadata<:loaded, "messages_users">,
 accepting_choices: false,
 all_choices: [%{"c" => 129138, "nc" => 129154}, ...],
 comparisons: [%{"a" => 129138, "b" => 129154},  ...],
 conversation: #Ecto.Association.NotLoaded<association :conversation is not loaded>,
 conversation_id: 1621, id: 132055,
 inferred_choices: [%{"c" => 129138, "nc" => 129154}, ...],
 manual_choices: [%{"c" => 129138, "nc" => 129130}, ...],
 message: #Ecto.Association.NotLoaded<association :message is not loaded>,
 message_id: 12748, rid: nil,
 user: #Ecto.Association.NotLoaded<association :user is not loaded>,
 user_id: 51959}

I don’t want to dig too deeply into why the states are what they are, but suffice to say that they have been well-researched and tested, and I don’t want to explain too much industry context

Now, we have been monitoring our app in production for a while, and noticed that, as the number of these processes alive increase, memory usage goes up almost exponentially.

With 600 MessageEngine.Users and 600 MessageEngine.Thoughts, we measured almost 35GB of RAM being used across the cluster.

I first tried to measure the amount of memory used just by the state of the process, but this doesn’t seem like nearly enough data to have that substantial of an impact.

I popped into observer to learn more, and ran the following tests:

30 users and 30 thoughts

With:
length(MessageEngine.User.all_choices) = 0
length(MessageEngine.User.manual_choices) = 0
length(MessageEngine.User.inferred_choices) = 0
length(MessageEngine.User.comparisons) = 0

One MessageEngine.User process was consuming 139kb of memory
One MessageEngine.Thought process was consuming 3kb of memory

With:
length(MessageEngine.User.all_choices) = 53
length(MessageEngine.User.manual_choices) = 20
length(MessageEngine.User.inferred_choices) = 33
length(MessageEngine.User.comparisons) = 53

One MessageEngine.User process was consuming 502kb of memory
One MessageEngine.Thought process was consuming 25kb of memory

300 users and 300 thoughts

With:
length(MessageEngine.User.all_choices) = 0
length(MessageEngine.User.manual_choices) = 0
length(MessageEngine.User.inferred_choices) = 0
length(MessageEngine.User.comparisons) = 0

One MessageEngine.User process was consuming 1089kb of memory
One MessageEngine.Thought process was consuming 6kb of memory

With:
length(MessageEngine.User.all_choices) = 53
length(MessageEngine.User.manual_choices) = 20
length(MessageEngine.User.inferred_choices) = 33
length(MessageEngine.User.comparisons) = 53

One MessageEngine.User process was consuming 4023kb of memory
One MessageEngine.Thought process was consuming 41kb of memory

So, as you can see, not only is memory usage per-process scaling up a lot just by adding ~50 maps to a list, the usage of each process also seems to be dependent on the number of processes alive! An order of magnitude increase in the number of processes results in an order of magnitude increase in the memory usage of each one.

This seems like really weird behavior to me, and I’m kinda stuck on where to go next, because, by my calculations, the memory usage of the state of these processes should be more like 20-50kb each (used this guide: Memory Usage — Erlang System Documentation v29.0.2).

Here’s a full dump of the state of a process that was using 4023kb of RAM: Memory usage · GitHub

Any help would be greatly appreciated.

33 comments

#otp #troubleshooting #genserver

1 14642 33

2019-11-06 00:30:57 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

sikanhe

Are you caching these users/thoughts inside your gen_server state? If so (someone correct me if wrong), due to immutability the reference to the old states/references are kept in memory. Which means, when you update the state, the maps are copied over and over and never garbage collected since gen_server is a living process.

The first way that comes to my mind to deal with this issue is to use ETS since it is a mutable structure.

Post #1

NobbZ

That’s the way bean works… If it has to increase size of a process heap it won’t shrink it until it really has to, eg because total memory wouldn’t be enough for all processes without shrinking some heaps.

Also there are many different ways to collect memory data of a process which did you use? If you really measured the total heap size it doesn’t say much since heap can shrink and grow.

Post #2

pdilyard

Hmm, interesting. I figured the unused old copies data structures would be garbage collected.

My use-case is kind of caching, but what really happens is that a user is loaded into memory in a process, then a whole bunch of changes occur to the structure in a 2-5min window, then the new state of the process is dumped back to the database.

The reason I’m not using ETS is because I need to distribute these across nodes in a cluster, and using Swarm as a process registry is a nice way to do that.

Post #3

pdilyard

I measured primarily using :observer.

Post #4

NobbZ

Which metric do we speak about?

Post #5

imetallica

@sikanhe you are somewhat correct. The process will only garbage collect when it exits/hibernates.

@pdilyard what you can do and I think it might work is: try to invoke on any of your callbacks (the mutating state one preferably) :erlang.garbage_collect/0. I think that might solve your problem on cost of some performance degradation.

Post #6

pdilyard

I used the “Memory and Garbage” section under “Process Information”.

And then also calculated what I thought the amount of memory usage should be (or there about) based on this guide: Memory Usage — Erlang System Documentation v29.0.2

Post #7

pdilyard

Shouldn’t the process be eventually garbage collected? I’ve let it sit for 10-20 minutes without any activity and the memory usage is still very high.

Post #8

NobbZ

There will be no GC if there is no reason to.

GC happens (simplified) only under 2 circumstances. Either stack and heap are colliding, so the current heap size will be doubled while still collecting garbage of the process. AFAIK this is the exact metric you are observing, just the amount of awailable heap for the process, used/filled or not.

The other reason why GC may kick in, is because another Process is OOM and BEAM tries to get more memory from other processes by collecting and shrinking them.

This article about GC in OTP 19 explains it pretty good and in all the detail you might or might not need.

Post #9

brightball

I’d agree with the others here. If you are going to update an in memory data structure, you are better off doing it in ETS if it’s getting updated pretty constantly. If it’s not, then you might be better off with mnesia to distribute it across the cluster. IMO unless the updates are happening almost constantly over the course of 2-5 minutes you are going to be better off persisting it and then retrieving it when the updates come.

Post #10

Load more posts (23 remaining)