When the CPU is overloaded the message queues backup but if CPU usage is okay message queues stay mostly empty.
256 connections per service per cpu. So if you are talking to two services you’d have two pools each with 256 max_connections. Calls to each service use their own pool. We were only calling two services heavily so if you are calling more you may need to adjust a bit.
Very interesting thread. Based on the description of incoming / outgoing request rate, you may want to read the guide my team has put together:
This came about due to RabbitMQ MQTT users who ran into issues in scenarios like you describe. You may get a lot of mileage out of adjusting system-level TCP settings.
I run a similar application that handles over 1B requests a day. It definitely sounds like you are hitting limits on TCP ports, either open ports in general or TCP ephemeral ports.
We use dedicated servers for this. It’s nice to get 24 CPU cores and 32 GB of RAM for about $100/month… Here are some things to look at. After that, it’s probably K8s related.
/etc/security/limits.d/foo-limitsfoo soft nofile 1000000
foo hard nofile 1000000
Open file limits for systemd (which doesn’t respect the limits file), e.g.
LimitNOFILE=65536
Open file limits in the VM, in vm.args for your release, e.g.
-env ERL_MAX_PORTS 65536
You may also want:
## Enable kernel poll and a few async threads
+K true
+A 128
worker_rlimit_nofile 65536;See Serving your Phoenix app with Nginx
Doing the math:
It hits you very hard running behind Nginx as a proxy, but can also hit you on the outbound side when you are talking to a small number of back end servers.
Tune the kernel settings to reduce the maximum segment lifetime, e.g.:
# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
# Recycle and Reuse TIME_WAIT sockets faster
net.ipv4.tcp_tw_reuse = 1
There are other kernel TCP settings you should tune as well, e.g.
sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
ulimit -n 20000000
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
sysctl -w net.core.wmem_max=16384
See The Road to 2 Million Websocket Connections in Phoenix - Phoenix Blog
If you are getting limited talking to back end servers, then it’s useful to give your server multiple IP addresses. Then tell your HTTP client library to use an IP from a pool as its source when talking to the back ends. So the equation turns into “source IP from pool” + random port + target IP + 80.
You may be able to reuse outbound connections, with HTTP pipelining, if the back ends support it.
At a certain point, the back end servers may be the limit. They may benefit from having more IPs as well.
DNS lookups on the backends can become an issue. We have had hosting providers block us because they thought we were doing a DOS attack on their DNS. Run a local caching DNS on your server.
See Best practices for deploying Elixir apps
Glad to give you more specific help if you need it.
Mother of god 
Also thank you @lukebakken
these replies are both 1000% phenomenal and I’m super grateful. I will have to go through each of these very slowly in detail, THANK YOU!!!
A few questions
-env ERL_MAX_PORTS 65536 the same as +Q 134217727?
ERL_MAX_PORTS is the same as +Q, use that
Ok thanks. I actually dont think GKE uses CoreDNS though, they chose to keep using KubeDNS as as far as i can tell
TLDR this is a game of whack-a-mole move-the-bottleneck. A lot of this advice is covered above but I added some additional tips and hopefully some useful commentary.
If you’re still struggling with this I would bypass all of docker/kubes and spin up a bare metal instance at https://packet.net/ they provide excellent short-term h/w and networking just like cloud VMs just on real hardware, and do a quick comparison to see what your app ends up with without the layers of virtualisation and load balancer goop in the way, and then in order of least effort/max benefit:
read the error messages, they tell you what’s wrong. all the logs, all the time. measure the impact of your changes.
ensure you are running with max ulimit
check your erlang VM ports with recon http://ferd.github.io/recon/ and monitor it to ensure you’re not running out. It’s an easy fix, and adding overhead is not expensive in memory terms.
ensure your socket listen queues are not backlogged in erlang. See http://erlang.org/doc/man/gen_tcp.html#type-option for some possible options to pass through to your HTTP/TCP libraries, but basically use large buffers, no nagel delay, use tcp keepalive. This varies depending on the particular library you’re using, but this should get you started at least. You will see from outside with netstat -Lan |grep <port> if you’re running out. A healthy acceptor should be 0/0/... at all times, anything else you have problems.
Here’s my haproxy and couchdb for example:
fffff80e098a17a0 tcp46 0/0/2000 *.443 (haproxy)
fffff8071a26e000 tcp4 0/0/2000 127.0.0.1.5984 (beam.smp from couchdb)
tune tcp ephemeral port range, and tweak TIME_WAIT reuse if logs show you’re running out. I see sonewconn errors in kernel logs on FreeBSD but I’m not sure what linux spits out. The rabbitmq examples are excellent, https://www.rabbitmq.com/networking.html more notes below. In particular I’d expect to see {error, eaddrnotavail} being returned from attempts to open new connections in the VM. These should bubble up to your HTTP layer.
get real data from the network side with tcpdump & wireshark to see how your external connections are behaving. You may need help interpreting these, if so just skip this til last.
ensure that you’re using persistent outbound TLS + HTTP1.1 (or better) connections to your upstream APIs. The first step is to use multiple IPs for outbound connections in your BEAM pool, and potentially secondly to move some work out of the VM entirely - I use haproxy and also do TLS termination there rather than inside the BEAM. This minimises the work the VM has to do, and also gives me a very nice stats page on how the connections are being handled, or retried, and additional load balancing capabilities across instances and servers.
go back and read your logfiles again hand in hand with proxy and debugging output looking for correlations.
Further notes below.
Personally I am a huge fan of grabbing a tcpdump on the server, and throwing that into wireshark to see what’s actually happening on the wire. Every time I’ve needed to do this, I’ve wished that I had done it earlier. I appreciate this is not necessarily everybody’s cup of tea but if you’re dealing with network issues, then you need to see what’s actually happening on the wire.
It’s really common (as others have pointed out) to run out of ports, both in the Erlang VM, and in the OS (called ephemeral TCP port exhaustion). The former is set with +Q as mentioned and just requires more ram, the latter is slightly more complicated.
If this is the case your OS should be logging this somewhere, you’ll need to research that to confirm if it’s happening. On my OS (FreeBSD) I see sonewconn errors in kernel logs. A symptom of this is increasing numbers of TCP connections in TIME_WAIT state. You can see this in dmesg and in output of ss -tan state time-wait as mentioned above. The next step is to make more ports available for use as ephemeral ports. See links for details http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html and https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/ have reasonable explanations.
For nertwork stack tuning in general, see https://fasterdata.es.net/host-tuning/ http://proj.sunet.se/E2E/tcptune.html & https://www.psc.edu/index.php/networking/641-tcp-tune are excellent references, as well as https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/performance_tuning_guide/#chap-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Networking but remember that you can’t simply tweak all the above settings without some consequences. slow gradual change and testing are required 
If you make your outbound pools too big, and you’re using persistent connections, the upstream servers can also start disconnecting idle connections, thus increasing your connection churn even further. If you can, contact those providers and ask them if they do any connection handling like this or not.
If you’re connecting over and over again to the same upstream IPs or all your connections are coming from a load balancer, (e.g. kubernetes, AWS LB) you may be hitting the time_wait state much much earlier than expected, as the BSD socket API uses the quad tuple (source IP, source port, dest IP, dest port) as a unique key. The solution is not to enable socket reuse in the kernel, but to inject further source & destination IPs into the mix. You can configure additional (non-public) IPs on the server, and have a load balancer bound to more IPs on both inbound and outbound legs of the proxy (if both are needed) so that the quad tuples are spread out across more IPs. Again the output of ss or netstat -Lan as a simple wc is not helpful, you need to see if a given IP is hitting your ephemeral port limits.
Finally, this (at least for me) is where haproxy shines, I can see what’s happening in the logs and change many of these network stack related settings without interrupting my BEAM applications. Separation of concerns is a nice thing, but you do introduce a further component.
Wow this is 1000% incredible, thank you. I’m literally bookmarking this reply for years to come.
Can you tell me how to run TCP dump in the way you do to feed it into wireshark? That sounds very helpful. I’ll need to read this a few more times to absorb it all. THANK YOU