Kubernetes + libcluster help (GKE)

erlang
deployment
kubernetes
#1

Hi,

I want to get my Elixir app to run with libcluster on Google Kubernetes Engine.

I found a similar article that’s somewhat related. This article talks about how to create a “headless service” that returns all the pod IPs, which he then passes into Peerage. You can search for myapp-service-headless to find his Kubernetes YAML file.

It looks like Peerage provides the same functionality

https://hexdocs.pm/libcluster/Cluster.Strategy.Kubernetes.html
https://hexdocs.pm/libcluster/Cluster.Strategy.Kubernetes.DNS.html#content (I think this is the one I need)

Unfortunately I don’t really understand the exact configuration with GKE + Libcluster.

I want to use the DNS A record method they mention in this doc (for some reason this doesnt match up with the docs with the same module name… maybe its out of date?). So far I DID get POD_IP and NAMESPACE properly passed into the docker container. So that step is good.

Here are my questions:

  1. I need a headless service right? It seems like the “DNS” method is preferable (the 2nd link). I tried making a headless service which matches that config in the article above, but it just says 0 pods and looks like it’s not working
  2. It says I should be able to run kubectl get endpoints -l app=myapp. I assume this is referring to the headless endpoint? Is that needed for both strategies?

Here is my headless service YAML (thats not working):

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-12-24T06:22:20Z
  labels:
    app: xyz-headless
  name: xyz-headless
  namespace: default
  resourceVersion: "251868"
  selfLink: /api/v1/namespaces/default/services/xyz-headless
  uid: 12345
spec:
  clusterIP: None
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: xyz-web ### this is actually my "deployment"... awsnt sure what to put here
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
#2

I’m doing a similar setup and this article helped me a lot https://engineering.dollarshaveclub.com/elixir-otp-applications-on-kubernetes-9944636b8609

repo: https://github.com/dollarshaveclub/ex_cluster

4 Likes
#3

Thanks! I’ll check it out

#4

I haven’t done this in a long time but found an old app that used it.

config/prod.exs

config :libcluster,
    topologies: [
    k8s: [
      strategy: Cluster.Strategy.Kubernetes,
      config: [
        kubernetes_selector: "app=${KUBE_APP}",
        kubernetes_node_basename: "my_app"]]]

vm.args

-name my_app@${MY_POD_IP}
-setcookie <thecookie>

looking through my service files I don’t see anything interesting so I’m not sure they are relevant other than setting app name.

deployment

kind: Deployment
metadata:
  name: my_app-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: my_app
    spec:

headless-service

apiVersion: v1
kind: Service
metadata:
  name: my_app-service-headless
  labels:
    app: my_app
spec:
  ports:
    - port: 8000
  selector:
    app: my_app
  clusterIP: None

service

apiVersion: v1
kind: Service
metadata:
  name: my_app-service
spec:
  ports:
    - port: 8080
      targetPort: 8000
      protocol: TCP
      name: http
  selector:
    app: my_app
1 Like
#5

dude awesome!!! THANK YOU!

#6

Just realised I forgot the deployment one.

metadata:
  name: my_app-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: my_app
    spec:
      imagePullSecrets:
        - name: regsecret
      containers:
        - name: my_app
          image: <my registry>
          ports:
            - containerPort: 8000
          args: ["foreground"]
          env:
            - name: HOST
              value: "app.myapp.com"

ingress (I think this has changed in newer versions though?) so probably not helpful.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my_app-appback-ingress
spec:
  rules:
  - host: appback.myapp.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my_app-service
          servicePort: 8080

Let me know if you get it working. If not I’ll see if any other config seems relevant.

1 Like