ellispritchard

ellispritchard

Performance of Erlang/Elixir in Docker/Kubernetes

Has anyone else observed throttling of Erlang or Elixir processes running in Kubernetes (or other Docker orchestration platforms)?

I’ve observed this in production a while ago (on a system I no longer have access to), but was unable to make headway. Originally, I noticed a wide variation of ‘ping’ times when connecting to a do-nothing endpoint, when the app was not under load, and then noticed throttling being recorded in Prometheus stats. I couldn’t seem to find a reason, nor create the problem in minikube etc. but then it was a very noisy/busy system, with 30-odd processes of various technologies and load per node.

Recently I came across what is possibly an explanation; this comes in two parts, one, a very detailed examination of CPU usage in the BEAM:

The second clue is talk about scheduler bugs in the Linux kernel, and what CFS quotas are supposed to do anyway:

https://github.com/kubernetes/kubernetes/issues/67577

It occurs to be that when running in a container under a CFS quota, we should all be setting +sbwt none to avoid this optimisation from throttling the Erlang process.

Anyone come across this? Thoughts?

Most Liked

garazdawi

garazdawi

Erlang Core Team

There are indeed scenarios when this is the case, however I would say that if you are not running alone on a machine, you do not want to have spinning enabled as you start to conflict with other services. I’ve been thinking about maybe changing the default to none or very_short, as that seems to be what most systems needs.

I can also see how changing +swct and +swt could be useful, as they adjust how eager the VM is to wake up more schedulers to help do work.

tristan

tristan

Rebar3 Core Team

Right. And note that you can change this at runtime, could be useful in experimenting.

If there are 8 cores and you assign the container N vCPUs you’d start the node with +S 8:N. Then adjust with erlang:system_flag(schedulers_online, NewN).

Would be neat to also see how having it automatically adjust depending on changes in load and allocated vCPUs would work out.

garazdawi

garazdawi

Erlang Core Team

Internal cleanup work by the schedulers. For instance, delayed de-allocation of remote memory blocks. otp/erts/emulator/internal_doc/DelayedDealloc.md at master · erlang/otp · GitHub

Tinkering with +swt would be the classic tradeoff of latency vs CPU. If you set it to very_high then you will use less CPU to do the work as the work tends to be co-located, so there is a lesser risk of lock contention and better cache usage etc etc. However, at the same time, the average time a job will wait in the run-queue before being allowed to run will go up, so you application will have higher latency.

I don’t consider them relevant at all. The default strategy is best for all scenarios. They exist in order to support the strange needs of some embedded systems run at Ericsson. +scl false is very similar to what you get when you run +swt very_low, so it could possibly be good when you want to optimize for latency.

Where Next?

Popular in Discussions Top

blackode
Elixir Upgrading is so Simple in Ubuntu and It worked for me Ubuntu 16.04 git clone https://github.com/elixir-lang/elixir.git cd elixir...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
CharlesO
Erlang :list.nth simple, but 1 - based nth(1, [H|_]) -> H; nth(N, [_|T]) when N > 1 -> nth(N - 1, T). Elixir Enum.at … coo...
New
ejpcmac
I have discovered Nix last month and I am currently on my way to migrating to it—both on macOS at home and the full NixOS distrubution at...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
marciol
Please, let me know if this kind of discussion already took place in another topic . Hi all, how do you consider if is better to build ...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
und0ck3d
Hello everyone! A few days ago I’ve created a topic here about how people were creating CMSs with Elixir and Phoenix. I’ve been studying...
New
RudManusachi
What configs will make sense to put to runtime.exs? – A bit of how I configure apps: I have generic configs in config/config.exs, dev...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29377 241
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New

We're in Beta

About us Mission Statement