Ecto SSL verification with Google SQL - 'connection requires a valid client certificate'

Hi,

I’m trying to set up client SSL validation with Google Cloud SQL, and things are not going well for multiple reasons.

First off, I was unable to get verify_peer mode working.

2022-08-02T18:42:38.029 app[7047257c] lhr [info] 18:42:38.028 [notice] TLS :client: In state :wait_cert at ssl_handshake.erl:2075 generated CLIENT ALERT: Fatal - Handshake Failure
2022-08-02T18:42:38.029 app[7047257c] lhr [info] - {:bad_cert, :hostname_check_failed}

I tried to add custom verify_fun: &:ssl_verify_hostname.verify_fun/3, through {:ssl_verify_fun, "~> 1.1"} but it falls over due to the bad arity:


2022-08-02T18:39:18.364 app[400cc581] lhr [info] 2022/08/02 18:39:18 listening on [fdaa:0:6924:a7b:276d:1:5d29:2]:22 (DNS: [fdaa::3]:53)
2022-08-02T18:39:20.347 app[400cc581] lhr [info] Reaped child process with pid: 567, exit code: 0
2022-08-02T18:39:23.215 app[400cc581] lhr [info] 18:39:23.214 [notice] TLS :client: In state :wait_cert at ssl_handshake.erl:362 generated CLIENT ALERT: Fatal - Internal Error
2022-08-02T18:39:23.215 app[400cc581] lhr [info] - {:unexpected_error,
2022-08-02T18:39:23.215 app[400cc581] lhr [info] {:badarity,
2022-08-02T18:39:23.215 app[400cc581] lhr [info] {&:ssl_verify_hostname.verify_fun/3, [[bad_cert: :hostname_check_failed]]}}}
2022-08-02T18:39:23.216 app[400cc581] lhr [info] 18:39:23.214 [notice] TLS :client: In state :wait_cert at ssl_handshake.erl:362 generated CLIENT ALERT: Fatal - Internal Error

so I ended up with verify: :verify_none, which works fine with OTP 24.

Next, upgrading to OTP 25 breaks things even further - with that, Ecto can’t find a client certificate.

2022-08-02T19:03:30.027 app[f5a13d3f] lhr [info] 19:03:30.027 [error] Postgrex.Protocol (#PID<0.1965.0>) failed to connect: ** (Postgrex.Error) FATAL 28000 (invalid_authorization_specification) connection requires a valid client certificate

My config:

 config :my_app, MyApp.Repo,
      ssl: true,
      ssl_opts: [
      verify: :verify_none,
        cacertfile: "/.../server-ca.pem",
        keyfile: "/.../client-key.pem",
        certfile: "/.../client-cert.pem"
      ]

In a similar security posture, post a credit card number here to get faster debugging help :stuck_out_tongue:

The symptoms you’re describing make me wonder if Erlang has the right certs in its trust root.

1 Like

I can drop a Monero wallet address if that counts :stuck_out_tongue_winking_eye:

Hm, maybe you’re right

Screenshot 2022-08-02 at 20.30.16

I’ve assumed that self-signed certificate should be handled by cacertfile option.

I have the same issue attempting to run verification: verify_peer and cacert file with cockroachdb’s cacert. Have yet to determine actual solution. I suspect we share the same problem and are looking for the same solution.

1 Like

There is some benefit to using TLS even with verify: :verify_none: a passive attacker who can only monitor, but not modify, network traffic will not be able to decode the traffic exchanged with the server.

Verification of the server certificate is necessary to protect against an active attacker. In a closed network environment like GCP some might argue that the risk of an active attacker is somewhat reduced, compared to connections that traverse the public internet.

If the server has a self-signed certificate, the certificate itself can’t be used to establish trust, in the same way that a CA-issued certificate can. But as long as the server continues to present the same certificate, or use the same key pair, you can ‘pin’ the certificate/public key and abort the connection if a different cert/key is used (which then requires manual verification to see if the change is legit or someone is trying to interfere).

The ssl_verify_fun package README has instructions on how to enable certificate or public key pinning. In Elixir that would look something like:

:ssl.connect('self-signed.badssl.com', 443,
  verify_fun: {&:ssl_verify_fingerprint.verify_fun/3, [check_fingerprint: {:sha, "36F81BA2C9B18032D8B7BC61B26F22F6086DAA95"}]},
  verify: :verify_none,
  reuse_sessions: false
)

(It doesn’t really matter here whether you pass verify: :verify_none or verify: :verify_peer - the verify_fun option overrides the default behaviors selected by that option anyway, but in recent OTP versions you’ll get a warning if you don’t set :verify option at all)

(Also note that you probably wouldn’t want to use reuse_sessions: false in production, but it is necessary while testing otherwise a full handshake may not be performed and your changes to the connection options won’t take effect, leading to surprising results)

3 Likes

Thank @voltone . I’ll try this. Great blog you got.

1 Like