If you are setting your limit to 4 on a 4 vCPU node, you could potentially starve out the underlying nodes from performing work.
We used to have some Ruby apps on the same hardware as Elixir apps without a limit set (oops). In a situation where Elixir was 100% slammed (8 schedulers at 100%), the Ruby app times jumped from 50ms to >30s per request. The only connection between the two apps was that they ran on the same node.
You’ve got this essentially right - the limits are applied through cgroups. The problems arise because, prior to an as-yet-unreleased version of Erlang/OTP, the runtime itself is not cgroups-aware. If you do not explicitly set the number of BEAM schedulers with +S, you’ll get a default number based on the physical core count of the Docker host, not based on the CPU shares dictated by the resource requests/limits. This means that if you’ve scaled up to many-core machines under the hood, you’ll get unnecessary contention when you get i.e. 16 BEAM schedulers fighting for 2000 millicores of CPU share.
Cannot endorse this enough! I need a very compelling argument to do otherwise, for anything that is user-facing or with response expectations measured in less than “minutes”.