benbonnet

Trying to understand Genserver memory leak

Getting started with otp; having a sample app which main purpose is to ingest data.

A genserver is subscribed to a given channel and receives load of data passed aroudn through pubsub. Its single purpose is tostore this data in the db (using ecto). It does not store any state.

  def init(_args) do
    Phoenix.PubSub.subscribe(PubSub, "CHANNEL_NAME")
    {:ok, []}
  end

  def handle_info({:matcher, data}, state) do
    upsert(data)
    {:noreply, state} # , :hibernate} added, solved memory leak, but ¯\_(⊙︿⊙)_/¯ 
  end

  defp upsert(klines) do
    Repo.insert_all(
      ModuleName,
      data, # array of struct
      on_conflict: :replace_all,
      conflict_target: [:key1, :key2, :key3]
     )
  end

Its memory invariably increase to some 100Mb, and never lowers. Googling around, found out about {:noreply, state, :hibernate}, and it solved my issue.

But no clue why this memory increase ended up to be permanent. Is it because of Ecto ? About the way the data is passed around (pubsub) ?
Hope the question is somehow relevant; would like to get a better understanding

regards

20 comments

#memory-leak

0 2565 20

2022-08-13 07:01:04 UTC

First 10 of 20 Posts!

ityonemo

I believe hibernate triggers a GC event.

You can also trigger GC manually using :erlang.garbage_collect/0

Post #1

benbonnet

It leads me be more accurate in my question; to try to understand when/how a genserver gets garbage collected

Or if is mandatory to manually manage a genserver’s memory

Post #2

ityonemo

It’s not required. but sorry, I don’t have answers for you on when the GenServer gets gc’d. Are maybe your messages coming in more rapidly than you can dispatch them to the database? (Aka do you always have a message in the message queue?). This is just a guess but I think if all of the messages are dispatched, the process goes into a tail call that calls GC, but if there are messages in the queue, the process is only put on ice by the scheduler.

Could also be some weird things with binary references, which are handled differently and have different lifetimes.

Here is specific data how the Erlang GC works, but that is not just for GenServers. Erlang Garbage Collector — OTP 29.0.2 (erts 17.0.2). If you’re curious you can always check out the source code for the gen module: otp/lib/stdlib/src/gen.erl at master · erlang/otp · GitHub + gen_server module otp/lib/stdlib/src/gen_server.erl at master · erlang/otp · GitHub

Post #3

benbonnet

thanks a lot for those links! and sorry did not meant to demand an answer, rather general advices

I should have mentionned, the genserver receives data every 15 minutes. It takes it something like 1 minutes to ingest the data; then it just hangs around not receiving anything, waiting for the next batch. Which leads me to think the GC was never triggered

Post #4

trisolaran

Sounds like this could be a plausible explanation: Extremely high memory usage in GenServers - #23 by sasajuric

Post #5

karlosmid

Hi! What helped in my team was to run function as a async Task:

Task.async(__MODULE__, : upsert, [klines])
    |> Task.await(:infinity)

In that case, when Task is done, gen server memory related to it is released immediately.

Heads up: this was in combination with when the gen server did repetitive tasks triggered by the timer.

Post #6

hst337

Well. it’s actually not a leakage, it’s just how the garbage collection works. I think you’re receiving big amount of structures (in the {:matcher, data}) structure, which is not collected right away

Post #7

hst337

That’s a bad solution, because it copies klines to the new process. The right solution is to trigger gc in the GenServer manually

Post #8

karlosmid

@hst337 what happens with klines in gen server after call to Task.async? Those are not garbage collected? Based on our testing, gen server memory usage immediately dropped.

Post #9

hst337

First of all, unused data will always be garbage collected. The question here is at which point it’ll be garbage collected
Second, here klines get copied into Task process you’ve created. So it might trigger gc in GenServer. But this klines will be in the Task until it dies.

But this is a bad solution, because you don’t need a process here, you don’t need to copy this data. You just need to call the Repo.all and then trigger minor gc

Post #10

Last Post!

benbonnet

so far no errors/missing data with the “working” approach - eg. dispatching through pubsub and centralizing the upsert in a single genserver; but am really wondering why such behaviour happens (the db timeout when each genserver was doing its upsert)