So this is the flag I have been thinking might have been causing me trouble, when running under CFS quotas with other processes.
For reference, see this design doc for the CFS: https://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt
TL;DR:
CFS’s task picking logic is very simple: it always tries to run the task with the smallest runtime value (i.e., the task which executed least so far). CFS always tries to split up CPU time between runnable tasks as close to “ideal multitasking hardware” as possible.
The theory is that the Erlang scheduler busy-looping is causing the CFS to record more CPU usage against the Erlang scheduler threads than they are using to do useful work in my code, and thus the Erlang scheduler thread gets throttled, to be fair to other threads e.g. if the quota is 100us of work, and if I do 50us of work in my code, before needing to wait on a socket, and there’s no other work for the Erlang scheduler to do, the busy-looping kicks in, and burns the remaining 50us for possibly no benefit, and thus the CFS thinks I’ve used my entire allocation, and throttles me, instead of allowing me to do more work shortly after…
So in my case, running Erlang in Docker, assuming the theory is correct, I would want to set +sbwt to none or very_short to avoid being throttled by the CFS, assuming that being throttled is worse than not immediately reacting to an event that might allow the Erlang scheduler to run my process.
If you are not being constrained in your use of CPU by a kernel scheduler, but trying to out-compete other processes on the same box, you won’t have the same rationalisation.
