What happens when Erlang VM / Beam runs out of memory?

Questions / Help
20

I’d go for:

  • Use :memsup as Jose showcased here: Get machine memory in elixir/erlang - #2 by josevalim. You could also use the same tool to make it fire alarms for you as @sasajuric already pointed out: Erlang -- memsup
  • Set a free memory threshold as a config value, e.g. if free/total memory ratio approaches, say, 90%, then you should have something in your pipeline that yields back-pressure.
  • You can also combine the above with Process.list() |> Enum.each(&:erlang.garbage_collect/1) but I’d advise against it because it could strain an already struggling system. Maybe just have a process that checks free memory every 5 secs and execute that code if the ratio is 80% or above – which reasonably maps to “a system marching to the limits but still having resources”.

Obviously the above is not at all a guarantee but it’s IMO a sane approach because it’s working with OS (or container) system limits. As a bonus, you can use the background process(es) that do monitoring to also send alarms / warnings via telemetry or various other dashboard systems, so you could interfere in time.

(Finally, and this could be a very random shot in the dark, and my apologies if so – if you have a load balancer in front of your service, just have it enforce a hard upper limit on request size; that’s a good way to make sure that the BEAM won’t spike in memory usage when a huge string is sent to it.)

2 Likes