Kalaww
Performance issue when running in Kubernetes
Hello, I have performance issue when running my elixir app in kubernetes.
I have a function that do a query to cassandra and them decode the result. The average execution time of the query is around 2ms and 1ms for decode, but when I do several calls, I observe sometimes a spike in execution time reaching 70 to 90ms either on query or decode step (but never the two of them for a same call).
I guess it is not related to network since it happened on the decode step too.
The cassandra connection is using a pool, and the decode step is run after releasing the connection.
I tried reproducing the issue locally, but it never happened (either with a local cassandra or the one in my kubernetes cluster). I am starting to think that maybe the way my app is running on the kubernetes node might have an impact. I don’t really know what I should do to have a better understanding of what causing these spikes.
If anyone has ideas, it will help me a lot.
I am using elixir 1.10
The app is running on a 4vCPU - 3.6Gb memory
I tried with these flags, without success
+sbwt none
+sbwtdcpu none
+sbwtdio none
Marked As Solved
benwilson512
You should set +S to the number of full CPUs allocated to your pod, minimum one. Setting 4 schedulers but only allowing 1 CPU is definitely going to cause contention.
Also Liked
dom
Are you setting the correct number of schedulers (+S)?
It’s explained under “Container Resources” in Kubernetes | Adopting Erlang
shanesveller
You’ve got this essentially right - the limits are applied through cgroups. The problems arise because, prior to an as-yet-unreleased version of Erlang/OTP, the runtime itself is not cgroups-aware. If you do not explicitly set the number of BEAM schedulers with +S, you’ll get a default number based on the physical core count of the Docker host, not based on the CPU shares dictated by the resource requests/limits. This means that if you’ve scaled up to many-core machines under the hood, you’ll get unnecessary contention when you get i.e. 16 BEAM schedulers fighting for 2000 millicores of CPU share.
benwilson512
I would read up on this: Configure Quality of Service for Pods | Kubernetes
For my core services I always aim for the Guaranteed QOS class, which means that the pod requests and limits must be identical. This does mean you may need some more nodes to ensure that you can actually provide the guaranteed level of resources. For secondary services Burstable is fine, and what’s left can get Best Effort.
Last Post!
shanesveller
Cannot endorse this enough! I need a very compelling argument to do otherwise, for anything that is user-facing or with response expectations measured in less than “minutes”.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"