luechtdev
Elixir / Erlang docker containers RAM usage on different OS's / Kernels
On my development VM (specs see below) - starting an elixir / erlang container takes up 2G of physical memory while running iex locally only takes up a few MB.
I only started an empty container with no modules loaded or code executed.
When more containers are started beyond the memory capabilities of the host, running Beam VM’s running in other (running) containers crash for the new instance to start.
Image of docker running with htop and system info (dev-vm)
observer_cli showing the memory usage
While memory usage is highly alarming on its own, other docker hosts are able to start the container with normal / reasonable memory consumption.
Production server running empty elixir container with normal memory usage
(Both instances running the same docker run -it elixir)
Other users could reproduce the problematic memory usage with the Ubuntu Desktop (kernel v5.15) but Ubuntu Server (kernel v6.4) is supposedly running just fine.
(Both running docker 24.0.2)
So far I tried:
- running different versions of the image
elixir:slim,elixir:alpine, … - limiting memory usage on docker → container crashes with errno 137 (out of memory);
- checking for swap → swap is disabled on both devices;
- running different versions of docker;
Any ideas on this problem would be apreciated since I’m unable to narrow down this problem to a Beam / Docker / OS / kernel level.
Kind Regards, Jannus
Marked As Solved
luechtdev
The problem seems to be the default Port limits visible under runtime_info/0.
In some cases (with excess memory usage) the ERL_MAX_PORTS value is set to 134,217,727 instead of the default 1,024. (approx. 50% of the max allowed value)
docker run -it -e ERL_MAX_PORTS=1024 elixir fixes the problem – in my case – but it still is just a workaround and doesn’t really explain this behavior.
Also Liked
everte
Thank you very much for this clarification and the link to the Erlang documentation!
It didn’t immediately click for me that ERL_MAX_PORTS is the same as the file discriptors ulimit -n.
With this information we know how to fix the issue, but it did not explain why it only happens on some systems and only when running elixir/erlang in a container. Then I stumbled upon this pull request for containerd: https://github.com/containerd/containerd/pull/7566
From this discussion it’s clear that at some point/version the file discriptor limits for containerd (and something also changed with systemD) changed from a real limit (1_048_576) to unlimited. With unlimited the mentioned +Q will take it’s maximum allowed value (134_217_727).
From that discussion it’s clear that having unlimited number of file descriptors is affecting other software as well.
In my opinion the Erlang numbers are reasonable enough (even with the maximum number it will still work fine, just use 1GB of memory), but I hope containerd will revert back to a “normal” limit. As shown here, for most people it’s not immediately intuitive what the cause of this excessive memory usage is and how to resolve. If your software needs (very) high limits of file descriptors, you’ll know and have the knowledge to manually override the settings/flags for it.
garazdawi
mjm
Thank you all, I’ve been losing my mind trying to figure out why my app suddenly needed 2GB of RAM when it wasn’t even close before. Turns out my Docker version got upgraded and I was suddenly hitting this issue. Setting ERL_MAX_PORTS fixed it!
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"