We dont have a reverse proxy, requests hit cloud flare which is just doing dns resolution, which resolves to aws load balancer where the vm is defined as the target. I only see a deregistration delay of 300 seconds defined for alb.
The database is an aws RDS postgresql and for it the cpu utilization for the instance hits 50% at peak and the write iops hits 6000. When I reindexed the tables the write iops came down. will try to monitor the server performance today to verify if I need to do autovaccum and reindex regularly
I’ve had it in the past where an AWS load balancer has not been working correctly and would return 504s depending on which node it hits. I’ve also had load balancers failing to serve HTTP/2 depending on the node hit. The point is sometimes these things break and it’s not typically somewhere you’d look. I’d recommend creating a new load balancer and seeing if you still have the issue.
I noticed that when I do sql inserts around 20 rows all in one go using db client on a highly used table I start seeing the timeouts on the client, I do VACUUM FULL on the table but that does not seem to stop it
