mekusigjinn

mekusigjinn

Hello, dumbfounded person here trying to get some insight into phoenix.

What would be having the endpoint hang for 13+ seconds until controller functions are run?

Context is this happens under heavy load.

It’s under kubernetes, distributed with six replica pods, plenty of CPU/Ram. but…
when the hanging happens, we see a lot of memory spikes.

We also noticed possible runaway creation of children by a DynamicSupervisor, so our thought is that high memory, or near - OOM status will make something in Phoenix wait until the controller functions are run.

On high memory: service usually works around 1GB, but when it starts to peak we saw it spike upwards of 6 GB and climbing until it goes into OOM and dies.

Can anyone shed some light into this? I think there’s some backoff logic but can’t really figure out what does it.

We put traces at each step so we know for sure that none of the controller function is running while it hangs for 13+ seconds.

Thanks in advance!

Showing Posts 1 to 10

dimitarvp

dimitarvp

You haven’t given any details so hard to say, but maybe big JSON payloads are being sent to your endpoints?

mekusigjinn

mekusigjinn OP

Thank you so much for replying back! Really appreciate your question.

Well we do have some endpoint requests with ~500+ UUID (in string form) in JASON object, in a list of strings format.

We don’t really have a lot of those calls though.

Are there anything else in particular that you’d like to know more?

At any given time during peak hours we get around 2.3K API calls, and rate is around 12 requests/sec.

dimitarvp

dimitarvp

Sadly it’s hard to guess just like that. I’d think something deserializes JSON to Elixir structs and is not properly releasing memory.

I can only shoot in the dark but I’d recommend you to get rid of that and work with JSON as raw string-keyed maps as a start.

But it’s not even certain that’s the issue. Your first step is to add some metrics / telemetry to your Plug-s.

mekusigjinn

mekusigjinn OP

We do use raw string-keyed maps, and that’s what we pass down in context functions, like “vanilla” context implementations created in phx.gen.* .

When you said adding metrics in Plug-s, can you elaborate a bit further? You mean the steps in MyApp.Endpint module, before hitting the router file?

dimitarvp

dimitarvp

That and everything else you can find really. Have a root OpenTelemetry span that starts somewhere in the router or whatever first-line-of-defense Plug is more relevant to you, and emit sub-spans and/or events downstream. Inspect your APM system and you should find who and where is waiting for so long.

BTW if memory serves Ecto et. al. are integrated with OpenTelemetry so you should check it out and make sure all these spans are being emitted and shown in your APM UI.

mekusigjinn

mekusigjinn OP

Yup I’ll do that. Thanks!

dimitarvp

dimitarvp

Sorry I can’t be more helpful but nothing else jumps to mind. I’d just start timing stuff whenever I can and isolate the culprit.

I am mostly sure this is something related to DB pools but have no basis to claim it.

mekusigjinn

mekusigjinn OP

Thanks for thinking about it with me.

For checking DB pools - what we can do is maybe lower the pool size per node right?

dimitarvp

dimitarvp

No, more like increase the queue_target and queue_timeout as a start. Maybe in conditions of higher load whoever needs DB connections has to wait more. These two options have helped me in very limited cases.

Link to docs.

mekusigjinn

mekusigjinn OP

Today our team was able to reproduce this locally.

Seems running

:recon_alloc.set_unit(:megabyte)
:recon_alloc.memory(:allocated_types)

during the peak revealed a lot of eheap_alloc to the tune of 40 GB and increasing fast. On re-runs I was able to get near 100GB (on M1).

binary_alloc stayed constant at ~125 MB

Would this shed some light on what’s happening?

Even during peaks the highest memory pid has only 253MB at most, but that ehap_alloc… my gosh it was growing incessantly.

I researched and it seems like this is due to long-running processes and GC being triggered, but I know very little about this / or how to break up “monolithic” processes

Where Next? Top

Trending in Questions Top

itzmidinesh
Hi all, I’m working on a Phoenix project where I need to make external HTTP requests and I’m trying to choose between Req and HTTPoison....
New
yarrichar
I’m very new to Elixir, so sorry if this is a silly question: I am wondering what happens to the mailbox of a process when the machine i...
New
Hellowman
Hello everyone! I am new to coding with Elixir/Erlang. And i am looking for some utilities for fuzzing testing like AFL++. Does anyone ...
New
pragdave
How do I tell Hologram not to escape HTML when substituting into templates? {Textile.render(@product.long_description)} escapes the HTM...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
rlipscombe
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
saleyn
Is there a way to import record definitions in the iex shell for pretty printing (similar to the rr command in the erl shell)? This woul...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews