I came across an interesting paper by Ulf Wiger (a very knowledgeable member of the Erlang community) about implementing overload protection in Erlang.
https://www.researchgate.net/publication/221211350_Generic_load_regulation_framework_for_Erlang
He also developed a library to embody the ideas in the paper:
And another person made a different library to do a similar thing:
The paper is a very succinct survey of the different approaches to dealing with the problem and I wonder if people here have had any experience with either the above libraries or applying some of the principles.
9 Likes
I one wrote a job scheduler using Genserver and :queue which did most of the things written in the Readme of the first linked Git repo.
However, that was high level; working for downloading (rate limit per domain) and extracting packages (hence the check for system resource pressure). Feedback from download jobs would adjust the rate of which the download jobs were executed; so hitting a limit would slow down the operations for a certain domain. Feedback in the scheduler for extract jobs would slow down the extraction rate.
On exit it saved the state (failure rate, rate limit etc) so on restart the ‘learned’ rate limits were applied per process again.
Being generic, it was tested by simple load generating jobs and could handle millions of tasks. (also: hammer a Mac OS network stack and it will start losing connections and/or packets, so that could be tested too without mocks…)
It worked very well; ran stable for years even when non-BEAM processes misbehaved.
3 Likes
