Elixir rest api returning 504 Gateway Time-out

Would need more information. I’ve seiged my server with over a 30k requests per second on 2-cores without an incorrect result (and after that it just slows down a lot more but still no incorrect, dropped, or invalid responses).

If it’s a 504 Gateway Time-out that sounds like there is a front-loader/gateway in front of Elixir and it is what’s getting overwhelmed (perhaps it has only a single connection open to the backend server?).

What is the gateway program, what defines ‘high load’, what is the gateway’s configuration for the reverse proxy, what are the hardware specs, etc… etc…

All requests go through cloudflare,if I don’t go through cloudflare and hit the server with ip directly even the welcome page shows 504 intermitently. This is on c4xlarge instance on ec2. It has 16 vcpus. How do I find the number of connections defined for backend (do you mean database) ?

In the config I see that the adapter is Ecto.Adapters.Postgres with pool size of 50. The maximum requests per second requests might be 3k requests updating the db and max 500 requests per second reading the info from rest api

I am maintaining the code some else has written, I can read the code and understand what process is doing. Let me know if you need additional information.

When you hit phoenix directly, it should never ever give you a 504, as it not acts as a gateway…

I just verified the config, the requests are going through amazon loadbalancer with target instance defined on port 4000. Requests hit cloudflare which has mapping to amazon loadbalancer which then forwards the request to the target instance

Do you have any issues, when you hit your application without any MITMs?

Do you have any issues when you hit your NGINX without any MITMs?

Do you have any issues when you use some MITM CDN?

Do you use free or paid plans?

My current experience with CDN providers like CloudFlare is to simply avoid them. They add hard to debug overhead without much benefit. Much more important is to have the servers “near” to the expected clients.

the clients are mobile apps updating data and also mobile and web clients accessing data from phoenix. We have paid plans

It’s still good to test each layer as that will tell you where the failures are happening. It’s extremely unlikely to be a fault of Elixir as this is the kind of work that the BEAM is designed for, it doesn’t fault in these ways, and it doesn’t return 504’s unless the program itself explicitly does so, so it is very likely to be one of those front gateways, so need to test each in turn.

I am currently checking with cloudflare support to check at their end. when I check top on the vm, I see beam is consuming 590.7 %cpu, is this normal on 16vcpu instance ?

Using 6vcpu’s worth on a 16vcpu is definitely not saturated so the BEAM would be responding just fine then.

The overall usage is dictated primarily by the application code though, so that could be normal or high all depending on the code. But considering it’s not hitting all 16 vCPU’s then it’s definitely not hitting it’s limits yet.

However, for a usual webapp that’s pretty high CPU usage so either it’s doing a lot internally or it’s really getting hit by potentially hundreds of thousands of connection.

For getting usage ‘inside’ the the built-in :observer module via :observer.start() is great for that if you can X over a GUI connection or so (or connect to it remotely if it’s exposed or can be exposed over, say, ssh or so), or if in a pure text shell you can get a reduced set of information by just opening a shell into the VM and running :etop.start() to show the top actors/processes.

However again, getting a 504 is entirely not normal from a standard Phoenix-built app, that still sounds like one of the gateways/load-balancers is failing and it would be good to bypass them at least for a time as a test.

I enterd this after starting the shell, but do not see any info. The clients post their locations every 5 seconds and there are other clients accessing this location information. the post api saves the lat,lon in the postgres db and the other mobile and web clients request lat, lon info using get apis