Thought a picture might help (and could be interesting for readers ):

071201×1057 93.6 KB

The yellow box is where the nightly quantum job happens. The label underneath each graph is the point of deployment of the new build without the ERL_COMPILER_OPTIONS='native' env var set. As you can see, since this deploy the load-arg has been lower than previous. Thoughput is down, but it tends to roll in waves, so another 5hrs should see it peak again. I’m looking forward to seeing whether there’s another memory incident with the flag disabled.

Unfortunately there are no magic env vars like that. From your description, though, it seems to me like maybe you have an endpoint that is very memory-hungry. My usual suspects to look for would be loading a lot of data from the database with something like Repo.all on a huge table.

there is the fullsweep_after or ERL_FULLSWEEP_AFTER erlang — OTP 29.0.2 (erts 17.0.2) (can’t deep link for some reason..)

But I would be very careful using that, to the level of warning against using that.. it’s treating the symptoms rather than the problem.. (and can create a whole lot of other problems.. increased cpu/latency - unpredictable latency etc etc)

what is the nature of your system? do you have periodic cron/quantum jobs that does the spike.. is it one user that have obscene amount of data etc etc.

so identify what process/code is responsible for the spike..