ellispritchard
Performance of Erlang/Elixir in Docker/Kubernetes
Has anyone else observed throttling of Erlang or Elixir processes running in Kubernetes (or other Docker orchestration platforms)?
I’ve observed this in production a while ago (on a system I no longer have access to), but was unable to make headway. Originally, I noticed a wide variation of ‘ping’ times when connecting to a do-nothing endpoint, when the app was not under load, and then noticed throttling being recorded in Prometheus stats. I couldn’t seem to find a reason, nor create the problem in minikube etc. but then it was a very noisy/busy system, with 30-odd processes of various technologies and load per node.
Recently I came across what is possibly an explanation; this comes in two parts, one, a very detailed examination of CPU usage in the BEAM:
The second clue is talk about scheduler bugs in the Linux kernel, and what CFS quotas are supposed to do anyway:
https://github.com/kubernetes/kubernetes/issues/67577
It occurs to be that when running in a container under a CFS quota, we should all be setting +sbwt none to avoid this optimisation from throttling the Erlang process.
Anyone come across this? Thoughts?
Trending in Discussions
Other Trending Topics
Chat & Discussions>Discussions
Latest on Elixir Forum
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security










First Post!
tristan
At least one of our project at work (which runs on k8s) saw a large benefit from enabling
+sbwt none. But I don’t know that it is a tweak to make by default.Another cpu related configuration to watch out for when running on something like Kubernetes is your number of active schedulers. You can end up with too many schedulers for the cpu share allocated and end up wasting cycles on dealing with schedulers and have many in busy wait.
Most Liked
garazdawi
There are indeed scenarios when this is the case, however I would say that if you are not running alone on a machine, you do not want to have spinning enabled as you start to conflict with other services. I’ve been thinking about maybe changing the default to
noneorvery_short, as that seems to be what most systems needs.I can also see how changing
+swctand+swtcould be useful, as they adjust how eager the VM is to wake up more schedulers to help do work.tristan
Right. And note that you can change this at runtime, could be useful in experimenting.
If there are 8 cores and you assign the container N vCPUs you’d start the node with
+S 8:N. Then adjust witherlang:system_flag(schedulers_online, NewN).Would be neat to also see how having it automatically adjust depending on changes in load and allocated vCPUs would work out.
garazdawi
Internal cleanup work by the schedulers. For instance, delayed de-allocation of remote memory blocks. otp/erts/emulator/internal_doc/DelayedDealloc.md at master · erlang/otp · GitHub
Tinkering with
+swtwould be the classic tradeoff of latency vs CPU. If you set it tovery_highthen you will use less CPU to do the work as the work tends to be co-located, so there is a lesser risk of lock contention and better cache usage etc etc. However, at the same time, the average time a job will wait in the run-queue before being allowed to run will go up, so you application will have higher latency.I don’t consider them relevant at all. The default strategy is best for all scenarios. They exist in order to support the strange needs of some embedded systems run at Ericsson.
+scl falseis very similar to what you get when you run+swt very_low, so it could possibly be good when you want to optimize for latency.Last Post!
ellispritchard
Thanks @mattbaker glad to find it’s still relevant (and helpful!)