SSL issues connecting to Heroku

Erlang & BEAM Forum (archived) Questions / Help (archived)
, ,
1

Hi!

We are experiencing some issues connecting from Erlang to a Heroku app (which at the end doesn’t matter, the issues is that we could not debug it at the time).

We are using:

  • Erlang 22.3
  • Elixir 1.10.2
  • :gun library

However, as demonstrated on the examples below, this issue is also happening by calling Erlang :ssl module directly.

To test the issue, I have deployed a base Phoenix app to Heroku, which can be found on: https://github.com/odarriba/phoenix-heroku-example

and can be accessed both in:

The issue we are having is that when we connect through the custom domain, it produces an Internal Error on Erlang’s SSL module:

Accessing through the custom domain

iex(3)> :ssl.start()
:ok

iex(4)> sock = fn() -> {:ok, s} = :gen_tcp.connect('ssl-bug-example.oscardearriba.com', 443, []); s; end
#Function<21.126501267/0 in :erl_eval.expr/5>

iex(5)> :ssl.connect(sock.(), [])

16:37:54.462 [info]  TLS :client: In state :hello received SERVER ALERT: Fatal - Internal Error

{:error,
 {:tls_alert,
  {:internal_error,
   'TLS client: In state hello received SERVER ALERT: Fatal - Internal Error\n '}}}

Accessing through the herokuapp.com subdomain

iex(6)> sock = fn() -> {:ok, s} = :gen_tcp.connect('ssl-bug-example.herokuapp.com', 443, []); s; end
#Function<21.126501267/0 in :erl_eval.expr/5>

iex(7)> :ssl.connect(sock.(), [])
{:ok,
 {:sslsocket, {:gen_tcp, #Port<0.8>, :tls_connection, :undefined},
  [#PID<0.144.0>, #PID<0.143.0>]}}

Both tests are with Erlang 22.3 compiled on OS-X, but we are seeing the same issue in Linux.

To debug it, we tried to see the TLS protocol version and the cipher used on both endpoints - and both are the same.

Any idea what can be happening here?

1 Like
2

Since you are passing a TCP socket to :ssl.connect/2, rather than a hostname (to :ssl.connect/3), the ssl application has no idea what is the hostname of the server you are connecting to. It therefore sends a ClientHello message without a ServerNameIndication (SNI) extension. The Heroku endpoint at the IP address ssl-bug-example.oscardearriba.com resolves to is apparently configured to require SNI, so it can serve up the correct custom certificate. Since the extension is missing, the server sends and alert and closes the connection.

When you connect to the ssl-bug-example.herokuapp.com endpoint, SNI is not required, because Heroku just defaults to sending their own wildcard certificate here.

You can resolve the error one of two ways:

  1. Add server_name_indication: 'ssl-bug-example.oscardearriba.com' to the options in :ssl.connect/2, or
  2. Use :ssl.connect/3 rather than opening the TCP connection yourself, so SNI is sent automatically

Either way, don’t forget to add the additional options necessary to enable server certificate verification, otherwise the client will ignore the server’s certificate, enabling MitM attacks.

2 Likes
3

That was the right direction to solve the issue!

As I said on the first post, we are using :gun library for some internal process. The big issue looks like that when the library is connecting through a proxy server, it does not pass the SNI configuration to :ssl by default, and it should be passed through the options.

Also, at some point :gun splitted the transport_opts option into tls_opts and tcp_opts. Passing server_name_indication: 'HOSTNAME' inside tls_opts option made it work.

I have also added peer verification with a bundled CA cert as you suggested on the link.

Thanks a lot! <3

4

I have been thinking during the night and there should be something else under the hood.

The same code (without specifying the server_name_indication: option) was working nice when pointing to other providers that also has SNI for our custom domains (a GKE cluster with multiple LetsEncrypt certificates).

This is probably some different behaviour in Heroku side. Not even calling a Cloudfront-enabled domain - which also uses SNI - made that error arise.

The SNI specification solved the issue, I just don’t know why it happened only with Heroku (as far as I tested).

It’s a very particular case - connecting to a custom domain in Heroku through an HTTP proxy using :gun library. So it’s not easy to reproduce and debug.

Thanks a lot for your help @voltone !

5

It is Heroku’s choice to host custom domains on a separate endpoint and require SNI there. I guess those other services you mention simply fall back to sending the service’s wildcard cert. This would trigger a hostname match warning with most clients, but many Erlang/Elixir clients do not verify the server’s certificate by default, so it is entirely possible that they connected without issues.

6

you made the point (again) :smiley:

openssl s_client -connect ssl-bug-example.oscardearriba.com:443

returned the same error, while on other host it returns any other certificate.

That make sense, i didn’t know that there was a no-send-certificate option anywhere :sweat_smile: . Thanks a lot, the explanation its really appreciated :hug:

1 Like
7

There isn’t, really: Heroku just configured/coded their systems to send an internal_error alert. If there was proper support for such a configuration, the alert (and the resulting error message in clients) might be more helpful, e.g. “required extension missing: server_name_indication”

1 Like