Flame_k8s_backend - a FLAME Backend for Kubernetes

No feedback at this point, but just wanted to say thank you! I was hoping this would happen quickly!

Very cool

Have you thought about using the cluster certificate to verify the tls connections?

Libcluster does this and it should be relatively simple to adapt: libcluster/lib/strategy/kubernetes.ex at 3f1afbdb9ec0929ed99d35e5f875e55f8cbdd851 · bitwalker/libcluster · GitHub

I am doing that in the connect function: flame_k8s_backend/lib/flame_k8s_backend/k8s_client.ex at main · mruoss/flame_k8s_backend · GitHub

Or is it something else you’re referring to?

Oh, I take it back. I misread the code. Why are you even offering insecure_skip_tls_verify then?

If the URL in the SA token is an IP (not a FQDN), hostname verification fails with :verify_peer as there is no way to verify the hostname in the cert. This is the case e.g. on my local Kind cluster…

I am very open to better ways of dealing with this, though.

So… I’ve created a PR that removes the insecure_skip_tls_verify option in favour of setting server_name_indication to :disable if KUBERNETES_SERVICE_HOST is an IP address (instead of a FQDN).

However, I’d really like a “security audit” on this. I think this is as safe as it can be. I mean… no SNI, no hostname check. So we might as well disable it automatically, no?

Then again, I was surprised to see even AKS (Azure) settting KUBERNETES_SERVICE_HOST to a FQDN if and only if you add an annotation to your pod!

Maybe I should do something like Erlang does for verify_none: Keep the option in place, but if it is not set and I’m setting server_name_indication to :disable, print a warning.

Opinions anybody?

Here’s the PR:

https://github.com/mruoss/flame_k8s_backend/pull/5

I’m not sure about the perfect solution here either. I’ll message @voltone in the ErlEF security WG if he has any suggestions.

Setting server_name_indication: :disable not only drops te SNI extension from the Client Hello message sent to the server, it also disables hostname verification altogether. So while the client still checks if the server is presenting a certificate that was issued by a trusted CA, it does not check if we have reached the server we intended to reach. That’s arguably better than verify: :verify_none, but I think we can do better still?

What identities does Kubernetes put in the certificate that the server presents, in the Common Name field of the Subject and in the SubjectAltNames extension? If the IP address appears anywhere and you connect with an IP address in the URL, then the default behavior of ssl (without :server_name_indication option) should be to try and match that IP.

One way to check what identities are being checked would be to pass the following :ssl option:
customize_hostname_check: [match_fun: fn a, b -> IO.inspect({a, b}); :default end]

Unfortunately I’m not fluent in Erlang. But I think this is actually a bug in Erlang’s public_key:pkix_verify_hostname/N function.

The certificate presented by the Kubernetes API Server contains the IP address (see the note on the Kubernetes docs.

I can verify that, looking at the id-ce-subjectAltName extension in the certificate:

{
  :OTPCertificate, 
  {
    :OTPTBSCertificate, 
     #…,
    [
     #…,
    {
      :Extension, 
      {2, 5, 29, 17}, 
      false, 
      [
        dNSName: ~c"localhost", 
        dNSName: ~c"hcp-kubernetes", 
        dNSName: ~c"kubernetes", 
        dNSName: ~c"kubernetes.default", 
        dNSName: ~c"kubernetes.default.svc", 
        dNSName: ~c"kubernetes.default.svc.cluster.local", 
        iPAddress: <<10, 0, 0, 1>>
      ]
    }
  }
}

Now the IP Address seems to be a binary. But looking at the Erlang code, I think it’s expecting a charlist, no? length() and list_to_tuple() are list operations, no?

This has been bugging me for so long now (I’m also maintaining the k8s library). If this could be fixed, it would be awesome. WDYT @voltone? I can also open an Erlang issue for this.

EDIT: Opened an issue: `public_key:pkix_verify_hostname/N` returns `{:bad_cert, :hostname_check_failed}` when connecting to IP addresses · Issue #7968 · erlang/otp · GitHub