Performance of Erlang/Elixir in Docker/Kubernetes

At least one of our project at work (which runs on k8s) saw a large benefit from enabling +sbwt none. But I don’t know that it is a tweak to make by default.

Another cpu related configuration to watch out for when running on something like Kubernetes is your number of active schedulers. You can end up with too many schedulers for the cpu share allocated and end up wasting cycles on dealing with schedulers and have many in busy wait.

Interesting. I actually tried running 2xvCPU schedulers to see if that helped: didn’t observe anything, but might have made it worse.

Yes, on dedicated machines, or outside containerisation, I believe it’s been demonstrated that +sbwt does help performance (which is presumably why it was added), at the cost of CPU/energy etc.

Helps only in some cases, otherwise it would be the default

What do you mean by 2xvCPU schedulers? As in if you specific 1 vcpu in k8s you used 2 active schedulers? That should hurt performance.

If you were running on AWS and using burstable performance instances you could be throttled because you exceeded your quota.

Certainly didn’t help, didn’t seem to make any difference though: theory was that having more threads would help it compete against Java processes running scores of threads.

These were m4.xlarge standard instances (4 vCPU), also tried m4.2xlarge (8 vCPU) for a while, so it wasn’t that.

If the VM would have exhausted its credits, the whole node would have practically ground to a halt in my experience. We experienced this on a Kafka cluster once, due to a bug in Kafka 0.8, not fun!

It’s probably not a great idea to run a busy production kubernetes node on a standard burst-able instance, without some sort of support for these in the k8s scheduler, since k8s basically tries to squeeze as much out of a VM as possible (kind of the whole point), so it might rarely earn credits; however, AWS T Unlimited instances allow you to pay to burst above your accumulated credits now, so there may be some use-cases where it makes sense.

By default it’ll look at the host system and spin up as many schedulers as there are logical CPUs, but depending on your type of load you can potentially get performance gains by increasing it. Do you know how many schedulers you’re running? It’s not an uncommon problem for VMs to make the assumption that it has access to all the host CPUs (Java etc), even if it is limited by eg cgroups. I’m not sure how well BEAM behaves here.

What was also interesting in the article about the BEAM CPU Usage was that even with disabling busy waiting with those three settings, they didn’t see a performance loss. In a cloud environment, when sharing CPUs or when limited by credits, it probably makes sense to disable it by default. Although benchmarking doesn’t hurt.

By default, it was definitely running the same number of schedulers as available cores (NB this is a system I don’t have access to any more).

I think what I’m interested in is finding out is what kind of analysis people have performed on the Erlang VM running in containerised platforms, and what the best VM settings are.

There are many years of experience of tuning the BEAM on dedicated hardware/machines, most of which probably translates pretty well to dedicated VMs, but running it in containers with CFS quotas, sharing a VM with numerous other containers, running multiple language technologies (i.e. a heterogeneous environment) , is relatively new, and we may not have figured out all the hitches.

Yeah, by default the BEAM is a pretty noisy neighbor. Compared to running some simple single threaded application, it’s harder to reason about performance and resource sharing in a cloud environment. It’ll gladly hog all CPUs because sharing CPUs or paying for cycles wasn’t necessarily a major consideration in its design. A bit like how Go aggressively allocates memory to avoid overhead, tradeoffs were made that don’t make sense in all use cases.

It might be valuable to actually look at reducing the number of schedulers to prevent the BEAM from over-spending on cycles. Even though the host system has some number of CPUs, if you’re limiting the application to say 1 vCPU, maybe you’ll get a behavior closer to the expected one if you also reduce the burstiness of the BEAM. If you only assign it 1/8th of the resources, it might not make sense to allow it to use 100% of them 1/8th of the time. Or it might, that’s up to your use case obviously!

Right. And note that you can change this at runtime, could be useful in experimenting.

If there are 8 cores and you assign the container N vCPUs you’d start the node with +S 8:N. Then adjust with erlang:system_flag(schedulers_online, NewN).

Would be neat to also see how having it automatically adjust depending on changes in load and allocated vCPUs would work out.