PSA: preventing outages due to DST Root CA expiry on Sep 30th

On September 30th, the root CA certificate DST Root CA X3 will expire. Let’s Encrypt have started rolling out a creative certificate chain to servers that use their certificates, which unfortunately may trip up some Erlang/Elixir applications once that DST Root CA expires. This could lead to outages due to failed TLS handshakes on the connection between such applications and those servers/services. Recent versions of Erlang/OTP include a fix that prevent those issues.

If you run an application on the BEAM that may communicate over TLS with one or more servers/services that present a Let’s Encrypt certificate, the safest thing to do is make sure the application is running on Erlang/OTP 23.3.4.5 or later, or 24.0.4 or later. If that’s not an option, check out this post for other, client and OTP-version specific recommendations.

Keep in mind that, due to client connection pools and TLS session reuse, an affected application may continue to function normally until some time after September 30th when the first new TLS handshake takes place…

Edit - let me clarify, this is not about servers (e.g. Phoenix apps) that have a Let’s Encrypt certificate, it is about any application that connects to a server with a Let’s Encrypt certificate, e.g. through an HTTP client, an MQTT client, a higher-level API client that uses an HTTP client, etc. That could include Phoenix apps, but also CLI tools, CI/CD scripts, embedded devices, …

54 Likes

You are a treasure, man. Thank you!

9 Likes

@AstonJ might be worth pinning?

5 Likes

…done :023:

6 Likes

Mint v1.4.0 was released, with a fix to ensure it connects successfully even on affected OTP versions.

10 Likes

@voltone I just want to say thank you for your stellar contributions around TLS in all the various ways you’ve helped the community over the years, including phx.gen.cert, blogs, and answering questions. This kind of stuff is incredibly valuable to everyone. Thank you!

25 Likes

@voltone Thanks!

Based on the posts, these libraries should work as-is, i.e. with no upgrades needed:

  • hackney
  • HTTPoison (because it uses hackney)

This one needs to be upgraded itself, but not Elixir/Erlang/OTP too

  • Mint

I can’t think of any other libraries off the top of my head.

True, as long as you rely entirely on Hackney’s default ssl options. If you pass in any ssl options of your own (e.g. to disable TLS 1.0 and 1.1, to send a client certificate, to change the ssl module’s log level) there is a good chance you will be affected. That is, if your custom ssl options do indeed verify the server’s certificate and you didn’t accidentally revert back to ssl's default of verify: :verify_none.

There’s a few other HTTP clients that make ssl options the responsibility of the user, including httpc and Gun. Here again it all depends on what options you pass in.

And that’s just HTTP clients. You have to consider other TLS clients as well:

  • Redis clients
  • MQTT clients
  • DB clients
  • SMTP (email) clients
  • App monitoring agents

And some people may have built some network protocol client of their own using ssl directly.

So unless you are absolutely sure that your application only ever uses Hackney with its ssl defaults, I would still recommend you upgrade your OTP version if at all possible.

3 Likes

Thanks!

Anyone know if there are alpine docker images in the works that will use Erlang 23.3.4.5? I’m not seeing anything on docker hub: Docker Hub

https://hub.docker.com/r/hexpm/elixir/tags?page=1&ordering=last_updated&name=23.3.4.5-alpine

1 Like

Sweet thanks!