ellispritchard

ellispritchard

Performance of Erlang/Elixir in Docker/Kubernetes

Has anyone else observed throttling of Erlang or Elixir processes running in Kubernetes (or other Docker orchestration platforms)?

I’ve observed this in production a while ago (on a system I no longer have access to), but was unable to make headway. Originally, I noticed a wide variation of ‘ping’ times when connecting to a do-nothing endpoint, when the app was not under load, and then noticed throttling being recorded in Prometheus stats. I couldn’t seem to find a reason, nor create the problem in minikube etc. but then it was a very noisy/busy system, with 30-odd processes of various technologies and load per node.

Recently I came across what is possibly an explanation; this comes in two parts, one, a very detailed examination of CPU usage in the BEAM:

The second clue is talk about scheduler bugs in the Linux kernel, and what CFS quotas are supposed to do anyway:

https://github.com/kubernetes/kubernetes/issues/67577

It occurs to be that when running in a container under a CFS quota, we should all be setting +sbwt none to avoid this optimisation from throttling the Erlang process.

Anyone come across this? Thoughts?

First Post! Switch mode

tristan

tristan

Rebar3 Core Team

At least one of our project at work (which runs on k8s) saw a large benefit from enabling +sbwt none. But I don’t know that it is a tweak to make by default.

Another cpu related configuration to watch out for when running on something like Kubernetes is your number of active schedulers. You can end up with too many schedulers for the cpu share allocated and end up wasting cycles on dealing with schedulers and have many in busy wait.

Most Liked

garazdawi

garazdawi

Erlang Core Team

There are indeed scenarios when this is the case, however I would say that if you are not running alone on a machine, you do not want to have spinning enabled as you start to conflict with other services. I’ve been thinking about maybe changing the default to none or very_short, as that seems to be what most systems needs.

I can also see how changing +swct and +swt could be useful, as they adjust how eager the VM is to wake up more schedulers to help do work.

tristan

tristan

Rebar3 Core Team

Right. And note that you can change this at runtime, could be useful in experimenting.

If there are 8 cores and you assign the container N vCPUs you’d start the node with +S 8:N. Then adjust with erlang:system_flag(schedulers_online, NewN).

Would be neat to also see how having it automatically adjust depending on changes in load and allocated vCPUs would work out.

garazdawi

garazdawi

Erlang Core Team

Internal cleanup work by the schedulers. For instance, delayed de-allocation of remote memory blocks. otp/erts/emulator/internal_doc/DelayedDealloc.md at master · erlang/otp · GitHub

Tinkering with +swt would be the classic tradeoff of latency vs CPU. If you set it to very_high then you will use less CPU to do the work as the work tends to be co-located, so there is a lesser risk of lock contention and better cache usage etc etc. However, at the same time, the average time a job will wait in the run-queue before being allowed to run will go up, so you application will have higher latency.

I don’t consider them relevant at all. The default strategy is best for all scenarios. They exist in order to support the strange needs of some embedded systems run at Ericsson. +scl false is very similar to what you get when you run +swt very_low, so it could possibly be good when you want to optimize for latency.

Last Post!

ellispritchard

ellispritchard

Thanks @mattbaker glad to find it’s still relevant (and helpful!)

Where Next?

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2976 91332 914
New
byu
@chrismccord : I just saw the Extract AGENTS.md from Phoenix.new into phx.new generator commit to the phoenix project. My initial shotgu...
New
arcanemachine
I was working on an Ecto migration and I needed a timestamp. So, for the nth time, I looked up the different data types for timestamps, a...
New
AstonJ
Just a general thread to post chat/news/info relating to AI/ML stuff that may be relevant for Nx now or in the future. Got anything to sh...
New
type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
matt-savvy
Is there a word for the ~> symbol used in Version strings? Do you also just call it a Squiggle Arrow™ ?!
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New
zachdaniel
Introducing AshStorage! Attachment and file management that slots directly into your resources :smiling_face_with_sunglasses: I had hope...
New

We're in Beta

About us Mission Statement