dsnipe

dsnipe

Memory leaking with long running processes

Hello.
I have an application in production under some load and it has a memory leak.
I’m preaty new in Erlang world, anyway I tried to debug it using recon library. And it looks like the problem is with binaries.
recon_alloc:memory(allocated_types) shows me that my binary_alloc and eheap_alloc increase all the time.
Tha application is quite simple.
At boot time my application loads some data from DB and run for each record a process (GenServer) with some state (not big). At one moment there might be from 1 to 10 such long running processes. I use Phoenix endpoints to listen HTTP requests and each go through all these long lived processes with request payload in format: %{"segments": [%{"some_var" => "some bn data"}]}. I suspect, that these binaries are not garbage collected, bevcause they still may have references from those long lived processes.
Migh be that a reason? And how to solve it?

Elixir 1.4, Erlang 19.3

Marked As Solved

sasajuric

sasajuric

Author of Elixir In Action

My advice is to simply give it a try and see how it fares. Adding hibernation is fairly trivial (just append :hibernate to each return tuple). And with hibernation in place you can quickly check whether the problem is fixed (if not, then the leak likely happens someplace else). If the hibernation is fast enough for you, than cool. If not, you’ll need to think of another way.

Also Liked

sasajuric

sasajuric

Author of Elixir In Action

That might indeed be a reason. A simple fix could be to hibernate each process after they do the work on the segment. So if it’s a GenServer, you can add :hibernate at the end of each tuple you return from handle_* callbacks (e.g. {:noreply, new_state, :hibernate}). This will lead to the GC of the GenServer and thus the references to binaries will be released.

Hibernation will cause some performance penalty, but whether that’s significant or not depends on the load on the process being hibernated.

Note that you can also use :recon.bin_leak/1 to try to find out the offending processes.

rvirding

rvirding

Creator of Erlang

It might be faster to just do a gc, :erlang.garbage_collect(), as doing a hibernate does more work because it first packs the process together then unpacks it on the next message. I don’t know if there is an Elixir call to this function.

NobbZ

NobbZ

Are you sure? AFAIK elixir can’t support OTP 16, since maps have only been introduced with OTP 17.

The README of the 1.4.0 tag of the github repository clearly states:

(Elixir requires Erlang 18.0 or later)

Also I think I’ve seen some kind of matrix once which desc ribed which version of erlang was supported by which version of elixir, but I can’t find it right now.


Aside of that, you can create space leaks (not memory leaks) easily when creating substrings from strings that are longer 64 byte. Therefore it is considered good practice to :binary.copy/1 strings you have extracted and want to use for longer than a few moments.

As a rule of thumb I do copy every string I pass into another function.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: The documentation above suggests that while ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New

We're in Beta

About us Mission Statement