Sure I could try but it was kind of a nightmare to deploy this. It might take me all day to deploy it w/o k8s, I will try though. I did see another google tutorial maybe I’ll try this: community/archived/elixir-phoenix-on-google-compute-engine/index.md at master · GoogleCloudPlatform/community · GitHub
I could probably do this fairly easily
Do you think I should try adding that DNS thing first before deploying to an instance?
You don’t necessarily need to deploy outside K8s, just find a way to disable k8s DNS lookups from those specific pods. That’s achieved by making sure that the resolv.conf file that ends up in each pod is just using a normal DNS server instead of your pod local one. I don’t recall how to configure that off the top of my head though sorry.
You don’t need to create new images just for debugging, what I suggest is connecting to an existing image and installing the packages for debugging there.
Based on what I’m reading it appears GKE still uses kube-dns
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
event-exporter-v0.2.3-54f94754f4-q6svn 2/2 Running 0 1d
fluentd-gcp-scaler-697b966945-6lslx 1/1 Running 0 1d
fluentd-gcp-v3.1.0-wr2lc 2/2 Running 0 1d
heapster-v1.6.0-beta.1-7988d7c5b5-9tfrs 3/3 Running 0 1d
kube-dns-548976df6c-bvxgg 4/4 Running 0 1d
kube-dns-autoscaler-67c97c87fb-x4n86 1/1 Running 0 1d
kube-proxy-gke-app-cluster-highcpu-64-ssd-afc9f592-zm9g 1/1 Running 0 1d
l7-default-backend-5bc54cfb57-hw4gv 1/1 Running 0 1d
metrics-server-v0.2.1-fd596d746-px56s 2/2 Running 0 1d
Ok I got ss -s working. Here’s when no problems:
bash-4.4# ss -s
Total: 2782 (kernel 88998)
TCP: 2683 (estab 511, closed 2167, orphaned 0, synrecv 0, timewait 2094/0), ports 0
Transport Total IP IPv6
* 88998 - -
RAW 0 0 0
UDP 0 0 0
TCP 516 76 440
INET 516 76 440
FRAG 0 0 0
I’m trying to get it to break so i can show you what it looks like then but that is easier said than done
This is what it looks like if I load it with more traffic than it can handle
bash-4.4# netstat -tpan | grep TIME_WAIT | wc -l
21750
and
bash-4.4# ss -s
Total: 3307 (kernel 90342)
TCP: 31988 (estab 993, closed 30990, orphaned 0, synrecv 0, timewait 30874/0), ports 0
Transport Total IP IPv6
* 90342 - -
RAW 0 0 0
UDP 0 0 0
TCP 998 16 982
INET 998 16 982
FRAG 0 0 0
I managed to deploy to a plain 64 CPU instance in amazon. I hand installed everything and ran it in foregroud in production mode via ssh.
Still same problem, although it got to about 50K total before dying, that is ~5-10% better but I can see when it starts to fail it looks like this:
