Yes that’s correct. It’ll still hit KubeDNS but it lowers the number of different domains that the KubeDNS service will try to resolve. All of my other comments about KubeDNS still apply though, this is just a handy optimization.
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
9mm
Hey everyone I’m going through these replies slowly, sorry if I don’t respond to you right away.
I am 1000% grateful and thankful for all your guys and gals input/feedback. I’m processing all of it currently, and it’s taking me awhile to research each reply
In my pod YAML this is what was there:
resources:
requests:
cpu: 100m
But thats the only limit
This is interesting though. I’m reading here: https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits
I originally wanted the LOWEST number of apps (because they all log independently and its going to create a database bottleneck if i have 100 things logging at once). If it HAS to come to that, so be it, but I’d like to avoid it.
What if I put it at 64000m? it seems to a bad idea according to this document because erlang may not know how to deal with that many CPU’s. does that sound accurate?
2
outlog
looks like you found the reason! that is 0.1 cpu per pod.. explains your test results.
read
and
1
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
Phoenix has been tested on boxes up to 128 cores and it scaled linearly, so 64 should be fine. I’d consider manually specifying the number of schedulers because I don’t know off the top of my head if erlang can tell what you’ve allocated for the pod or not. Usually you want twice as many schedulers as cores. So if you do 6400m CPUs then do +S 128 in your vm.args file.
The final main thing to note for the 1 erlang pod route is to figure out if any of the libraries you’re using have a pool of fixed size. You’ll want to make sure to increase its size.
@outlog requests and limits are different things. Doing 0.1 CPU requests doesn’t limit it to 0.1 CPU, but it does mean that it may be throttled up to that if other pods want CPU. The way to achieve the highest Quality Of Service guarantees in K8s is to specify BOTH an identical CPU limit and request value.
3
9mm
Ok I actually tried running it on the instance itself (via SSH) rather than using bash IN the container:
Ip:
Forwarding: 1
416024035 total packets received
2 with invalid addresses
415262118 forwarded
0 incoming packets discarded
761912 incoming packets delivered
416003450 requests sent out
390 outgoing packets dropped
1 dropped because of missing route
Icmp:
22 ICMP messages received
3 input ICMP message failed
ICMP input histogram:
destination unreachable: 10
echo requests: 11
echo replies: 1
11 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo replies: 11
IcmpMsg:
InType0: 1
InType3: 10
InType8: 11
OutType0: 11
Tcp:
48909 active connection openings
16003 passive connection openings
41 failed connection attempts
8270 connection resets received
15 connections established
761425 segments received
840810 segments sent out
225 segments retransmitted
0 bad segments received
2994 resets sent
Udp:
467 packets received
0 packets to unknown port received
0 packet receive errors
470 packets sent
0 receive buffer errors
0 send buffer errors
UdpLite:
TcpExt:
33 resets received for embryonic SYN_RECV sockets
2 packets pruned from receive queue because of socket buffer overrun
9925 TCP sockets finished time wait in fast timer
31097 delayed acks sent
2 delayed acks further delayed because of locked socket
Quick ack mode was activated 3 times
272044 packet headers predicted
160323 acknowledgments not containing data payload received
101516 predicted acknowledgments
TCPSackRecovery: 7
Detected reordering 321 times using SACK
1 congestion windows recovered without slow start after partial ack
22 fast retransmits
TCPTimeouts: 66
TCPLossProbes: 147
TCPDSACKOldSent: 3
TCPDSACKRecv: 152
9 connections reset due to unexpected data
5 connections reset due to early user close
TCPDSACKIgnoredNoUndo: 139
TCPSackMerged: 7
TCPSackShiftFallback: 302
TCPRcvCoalesce: 70011
TCPOFOQueue: 566
TCPAutoCorking: 9
TCPFromZeroWindowAdv: 1713
TCPToZeroWindowAdv: 1713
TCPWantZeroWindowAdv: 1078
TCPSynRetrans: 56
TCPOrigDataSent: 395019
TCPHystartTrainDetect: 2
TCPHystartTrainCwnd: 654
TCPKeepAlive: 6156
IpExt:
InMcastPkts: 6
OutMcastPkts: 10
InOctets: 359183513039
OutOctets: 717144072828
InMcastOctets: 570
OutMcastOctets: 730
InNoECTPkts: 479288616
InECT0Pkts: 1
1
9mm
Awesome thanks!! And btw I still havent read your other reply I know it was a really good reply (abou tthe schedulers and vm.args) so I’m saving it a bit to do some of these easier replies first.
This is such good info, thank you I am eternally grateful 
2
9mm
Awesome! I will read those – but just to be clear would it be OK to put it at 64000m? or even “no limit” (can I delete the entire resource line)?
I was testing different instance sizes and I know it wont even schedule a pod into a node w/o enough resources. It would be great to move the limit entirely so its just 1 app per node regardless of size
0
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
Well the goal right now is debugging, so let’s focus on that right now. Debugging is about making sure we create as few factors as possible. How many nodes are you running? To get the highest QOS within K8s you want to do equal requests and limits. I would probably do 3200 for both so that you can also allocate a CPU or two for KubeDNS. Then manually set +S 64.
0
chrismcg
I really don’t know enough to read that output I’m afraid. In the container you could try netstat -atn | grep TIME_WAIT | wc -l. This assumes that the output of netstat -atn is similar to my local linux box and grep and wc are there too. If they’re not you could copy the output and grep it locally. The idea is just to see how many TCP connections are around and see if that corresponds in some way to the number where you start seeing problems.
0