sdr

sdr

High memory usage when performance testing simple Rustler NIFs

I’ve been playing around with Rustler NIFs to get familiar with them and understand how they interact with Elixir.

While running some simple toy NIFS and benchmarking them to be impressed with the performance differences, I was surprised with :process-memory-high-watermark warnings and opening the observer showed that memory had ballooned up to a couple of GBs.

The code isn’t creating any long term data and simply sums the numbers from A..B so I was surprised to see the jump in memory. The create is built release mode and benched with Benchee and it’s unclear what’s causing the memory jump.

Code is as follows

#[rustler::nif]
fn add(a: i64, b: i64) -> i64 {
    let mut r:i64 = 0;
    for x in a..b {
        r += x;
    }
    return r;
}

rustler::init!("Elixir.FastMath", [add]);

Any hints on where to look or insights into what’s causing the high memory usage?

Marked As Solved

sdr

sdr

After building some elixir/erlang muscle this has nothing to do with NIFs but is caused by Benchee collecting a list of samples for each iteration.

The same memory usage can be seen with a no-op function:
Benchee.run(%{fast: fn -> :ok end})

For short function calls this results in a large list (order of millions) of samples which explains the the memory spike.

Also Liked

sdr

sdr

A minimal repo to replicate this can be found at GitHub - sross/niftest · GitHub

sdr

sdr

Can confirm that I’m still seeing the same behaviour with rustler 0.26.0 and benchee 1.1.0 on Erlang 1.14 with OTP 25 under WSL.

Memory spikes to just under 3GB and the drops back to 1.2GB. the drop back to normal levels at the 6 second mark is when I ran a manual garbage collect.

bdarla

bdarla

I experimented with the provided repo and here are the results in my machine (Intel MBP with latest versions of Elixir and Rust as well as the employed libraries, i.e. rustler v0.26 and benchee v1.1).
I am not sure whether the issue remains or not. If anyone could judge better than me, please, do so.

Where Next?

Popular in Questions Top

baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44778 311
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement