ok, did you try to test the API you are calling? Is it possible that they are not able to handle the load you are putting on them? I would use some outside tool like ab for example to verify that you are not getting the same timeouts.
ashneyderman
9mm
I’m sure its not them because of a few things:
a) they can handle millions of requests a second (they’re data APIs)
b) if it were the API then simply multiplying the number of running apps on a single node wouldnt get wildly improved results
0
outlog
afaik this should not be this low for a high concurrency server..
can you echo 1048576 > /proc/sys/fs/aio-max-nr or change in sysctl.conf and report back?
0
NobbZ
9mm
I will try. I just realized that Tesla uses a pool by default even though hackney by default doesn’t use a pool.
So this entire time its been using a pool which has thrown off my mental model. Now that I know it IS using a pool and failing, I decided to test the pool settings.
The default connections is 50 with a timeout of 15 second I believe… I tried setting it to 200 with a timeout of 1 second.
You can see a pool def helped. I suppose ill try to see the max pool i can get on a single app instance. When I did this before I remember getting decreasing performance past a certain number, I thought it was 200 or so.
1
chrismcg
That’s suspiciously close to 32K. What does cat /proc/sys/net/ipv4/ip_local_port_range show and have you tried increasing it?
0
9mm
On the 64 cpu server thats not kubernetes it is:
root@testcrap2:/home/info# cat /proc/sys/net/ipv4/ip_local_port_range
32768 60999
0
9mm
I dont really see a change at all, but i will remember to change these when i go bck to kubernetes too, thank you.
0
9mm
If this is for the filesystem IO I believe my app hardly uses the disk if that matters. In the google cloud the disk IO is several MB with like 20-100 operations a second
0
chrismcg
There’s a bit here about changing the port range: Linux increase ip_local_port_range TCP port range · ma.ttias.be
Curious if you tried the net.ipv4.tcp_tw_reuse setting I mentioned above?
1