What makes an API HTTP request to Phoenix hang until going through controller function code?

You haven’t given any details so hard to say, but maybe big JSON payloads are being sent to your endpoints?

Thank you so much for replying back! Really appreciate your question.

Well we do have some endpoint requests with ~500+ UUID (in string form) in JASON object, in a list of strings format.

We don’t really have a lot of those calls though.

Are there anything else in particular that you’d like to know more?

At any given time during peak hours we get around 2.3K API calls, and rate is around 12 requests/sec.

Sadly it’s hard to guess just like that. I’d think something deserializes JSON to Elixir structs and is not properly releasing memory.

I can only shoot in the dark but I’d recommend you to get rid of that and work with JSON as raw string-keyed maps as a start.

But it’s not even certain that’s the issue. Your first step is to add some metrics / telemetry to your Plug-s.

We do use raw string-keyed maps, and that’s what we pass down in context functions, like “vanilla” context implementations created in phx.gen.* .

When you said adding metrics in Plug-s, can you elaborate a bit further? You mean the steps in MyApp.Endpint module, before hitting the router file?

That and everything else you can find really. Have a root OpenTelemetry span that starts somewhere in the router or whatever first-line-of-defense Plug is more relevant to you, and emit sub-spans and/or events downstream. Inspect your APM system and you should find who and where is waiting for so long.

BTW if memory serves Ecto et. al. are integrated with OpenTelemetry so you should check it out and make sure all these spans are being emitted and shown in your APM UI.

Yup I’ll do that. Thanks!

Sorry I can’t be more helpful but nothing else jumps to mind. I’d just start timing stuff whenever I can and isolate the culprit.

I am mostly sure this is something related to DB pools but have no basis to claim it.

Thanks for thinking about it with me.

For checking DB pools - what we can do is maybe lower the pool size per node right?

No, more like increase the queue_target and queue_timeout as a start. Maybe in conditions of higher load whoever needs DB connections has to wait more. These two options have helped me in very limited cases.

Link to docs.

Today our team was able to reproduce this locally.

Seems running

:recon_alloc.set_unit(:megabyte)
:recon_alloc.memory(:allocated_types)

during the peak revealed a lot of eheap_alloc to the tune of 40 GB and increasing fast. On re-runs I was able to get near 100GB (on M1).

binary_alloc stayed constant at ~125 MB

Would this shed some light on what’s happening?

Even during peaks the highest memory pid has only 253MB at most, but that ehap_alloc… my gosh it was growing incessantly.

I researched and it seems like this is due to long-running processes and GC being triggered, but I know very little about this / or how to break up “monolithic” processes