On September 30th, the root CA certificate DST Root CA X3 will expire. Let’s Encrypt have started rolling out a creative certificate chain to servers that use their certificates, which unfortunately may trip up some Erlang/Elixir applications once that DST Root CA expires. This could lead to outages due to failed TLS handshakes on the connection between such applications and those servers/services. Recent versions of Erlang/OTP include a fix that prevent those issues.
If you run an application on the BEAM that may communicate over TLS with one or more servers/services that present a Let’s Encrypt certificate, the safest thing to do is make sure the application is running on Erlang/OTP 23.3.4.5 or later, or 24.0.4 or later. If that’s not an option, check out this post for other, client and OTP-version specific recommendations.
Keep in mind that, due to client connection pools and TLS session reuse, an affected application may continue to function normally until some time after September 30th when the first new TLS handshake takes place…
Edit - let me clarify, this is not about servers (e.g. Phoenix apps) that have a Let’s Encrypt certificate, it is about any application that connects to a server with a Let’s Encrypt certificate, e.g. through an HTTP client, an MQTT client, a higher-level API client that uses an HTTP client, etc. That could include Phoenix apps, but also CLI tools, CI/CD scripts, embedded devices, …
@voltone I just want to say thank you for your stellar contributions around TLS in all the various ways you’ve helped the community over the years, including phx.gen.cert, blogs, and answering questions. This kind of stuff is incredibly valuable to everyone. Thank you!
True, as long as you rely entirely on Hackney’s default ssl options. If you pass in any ssl options of your own (e.g. to disable TLS 1.0 and 1.1, to send a client certificate, to change the ssl module’s log level) there is a good chance you will be affected. That is, if your custom ssl options do indeed verify the server’s certificate and you didn’t accidentally revert back to ssl’s default of verify: :verify_none.
There’s a few other HTTP clients that make ssl options the responsibility of the user, including httpc and Gun. Here again it all depends on what options you pass in.
And that’s just HTTP clients. You have to consider other TLS clients as well:
Redis clients
MQTT clients
DB clients
SMTP (email) clients
App monitoring agents
…
And some people may have built some network protocol client of their own using ssl directly.
So unless you are absolutely sure that your application only ever uses Hackney with its ssl defaults, I would still recommend you upgrade your OTP version if at all possible.
We use AWS rds and redis services, should I worry? I mean, I’m still updating, but i believe aws has its own certificate so validating it shouldn’t be an issue, not sure tho
No, I wouldn’t think so. The issue is primarily with servers using a Let’s Encrypt certificate, although the root cause is a more generic limitation of Erlang/OTP’s certificate chain handling prior to the patch. There have been other cases of cross-signed certificates causing problems in the past, and there may be more out there somewhere. So upgrading is definitely your best bet when it comes to ensuring outbound TLS connections will work as expected, now and in the future.
