Issue with http clients and some https certificates

Questions / Help
, ,
1

Please help me debug the issue with https certificates, need some ideas what to check next

When fetching data using http clients, some https:// urls returning “Certificate Expired”, despite being perfectly valid, e.g. https://investing.com/, https://api.amplitude.com/ .
It looks like it doesn’t depend on erlang version, elixir version, http library version.

Base system is (a) OSX 10.14.5 (b) Docker, Alpine Linux

test app #1

elixir 1.9.4
erlang 22.3.4.1 (also 22.1.8)
mojito 0.6.4
mint 1.1.0
httpoison 1.5.0
hackney 1.15.2 (also 1.16.0)

iex(8)> Mojito.request(:get, "https://api.amplitude.com/")
[info] TLS :client: In state :certify at ssl_handshake.erl:1764 generated CLIENT ALERT: Fatal - Certificate Expired

{:error,
 %Mojito.Error{
   message: nil,
   reason: %Mint.TransportError{
     reason: {:tls_alert,
      {:certificate_expired,
       'TLS client: In state certify at ssl_handshake.erl:1764 generated CLIENT ALERT: Fatal - Certificate Expired\n'}}
   }
 }}

iex(9)> HTTPoison.get("https://api.amplitude.com/")
[info] TLS :client: In state :certify at ssl_handshake.erl:1764 generated CLIENT ALERT: Fatal - Certificate Expired

{:error,
 %HTTPoison.Error{
   id: nil,
   reason: {:tls_alert,
    {:certificate_expired,
     'TLS client: In state certify at ssl_handshake.erl:1764 generated CLIENT ALERT: Fatal - Certificate Expired\n'}}
 }}


iex(10)> opts = [ssl: [cacertfile: :certifi.cacertfile(), verify: :verify_peer]]
iex(11)> HTTPoison.get("https://api.amplitude.com/", [], opts)
# same error - CLIENT ALERT: Fatal - Certificate Expired

test app #2

elixir 1.3.4
erlang 19.3.6.5
hackney 1.15.0
httpoison 1.0.0

iex(6)> HTTPoison.get("https://api.amplitude.com/")
[error] SSL: :certify: ssl_handshake.erl:1609:Fatal error: certificate expired

{:error, %HTTPoison.Error{id: nil, reason: {:tls_alert, 'certificate expired'}}}
2

According to curl the certificate has expired for at least https://api.amplitude.com/. There was an issue over the weekend with an expired root certificate from Comodo/Sectigo CA for https://openexchangerates.org as well. Perhaps these sites are using the same certificate chain. Have you checked their support site?

kip@Kips-iMac-Pro cldr_dates_times % curl "https://api.amplitude.com/"
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
4 Likes
3

There are high chance your guess is correct.
Though on my system curl, firefox, chrome all think there are no issues (both OSX and Ubuntu)

Thanks!

4

That interesting - my curl output is from MacOS.

curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
5

Using https://www.sslshopper.com/ssl-checker.html#hostname=https://api.amplitude.com/ has the details and its indeed the Comodo and AdTrust certificates that have expired on May 30, 2020.

At this point I assume some clients are correctly validating the certificate chain and some are not.

image
image1144×634 62.6 KB

1 Like
6 
# OSX
curl --version                                                                                                    ┆master
curl 7.62.0 (x86_64-apple-darwin18.2.0) libcurl/7.62.0 SecureTransport zlib/1.2.11

Ubuntu 20.04 - no issues
Ubuntu 18.04 - no issues
Ubuntu 16.04 - server certificate verification failed

Also, tried providing fresh cacert.pem from https://curl.haxx.se/docs/caextract.html to httption/hackney - no changes in behavior, same error

7

It is unknown how good/actual sslshopper is.

Check my SSL Certificate | Is your SSL certificate valid and installed ? says there are no issues
Check SSL Certificate - GeoCerts says all good
etc

8

Totally fair and I’m definitely not a domain expert. I’m just going on the expiration dates on the certificates. So yes, I’m assuming they are correctly decoding the certificate metadata - and perhaps I shouldn’t.

This is where one usually invokes @voltone as the eventual source of truth :slight_smile:

9

I ran the check on https://ssllabs.com which I think is well respected. It shows the intermediate and root certificates sent by the server as expired.

However, it also shows that there are three paths to establishing trust, one that includes the expired certificates and two that do not. As I understand it from some reading, the :ssl app is unable to resolve alternative trust paths and therefore will fail to validate the peer in more situations than other clients. (I know that is not precise, mostly because I only understand a very small amount of this topic).

It also suggests why you are seeing different outcomes from different clients.

2 Likes
10

This basically means the root of the issue is how :ssl operates and therefore it should affect a lot of people, not just me. Also, it’s surprising given this thing exists for a dozen (or two) years already

11

Well to be fair, the server’s certificate chain is invalid and should be fixed.

And in addition, :ssl definitely has limitations and is a salutary example of why rolling your own is something I imagine the Erlang developers would probably avoid if they got to do it over again :slight_smile:

Actually Im being imprecise again. The function to verify peer is something a user provides - perhaps there is an enhanced validation function already out there in the cosmos.

12

This is an issue with older OpenSSL versions - we’re running 1.0.2g in production (:scream:) and encountered the same thing. Erlang depends on OpenSSL via libcrypto.so (at least on Ubuntu)

Long writeup here: https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-external-root-expiration-may-2020

2 Likes
13

I fixed the same issue this morning with the Amplitude API. I couldn’t see a way of fixing it at our end, so I ended up switching to https://api2.amplitude.com, which they set up over the weekend with a simpler trust chain. More details here: https://status.amplitude.com

Modern browsers support this type of multiple trust chain, so I’d have expected Elixir/Erlang to too. Is there an issue being tracked somewhere?

1 Like
14

If the webserver is IIS all you need to do is add a valid root and intermediate. Both of which expired same day.

https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT has a write up and links to get the current root and intermidiate. Comodo shares a few roots with Sectigo.

use MMC on IIS server to add these accordingly.

If TomCat or java process you’ll need to update the PFX containing the cert, you may also need to update the cacerts of java 6, 1.6 and lower. Java 7 and 8 are good, java 6 should be good.

You can look in cacerts with keytool -list -keystore…

cd 'C:\Program Files (x86)\Java\jre1.8.0_181\bin'
((((.\keytool -list -keystore c:\captures\cacerts -storepass changeit) -join "`n") -replace "(?sim)(?:\S{2}$)","`$0`n|") -replace "`n",'').split('|') | ? {$_ -match 'fingerprint'} | ConvertFrom-Csv -header 'cacert','date','dateYr','cf','fingerprint' | where fingerprint -match "02:FA|ea:fa|f5:ad|2b:8f|a7:9e|d1:eb"
15

I don’t think it related to erlang’s :ssl (and http clients) - as far as i understand erlang uses it’s own code to process ssl stuff. Also, tried to update system to 1.1.1g just in case - no changes so far

16

Exactly. The :ssl application treats the certificates sent by the server as a linear, unbreakable chain, ignoring any self-signed certificates that may have been sent because of misconfiguration. (Recent versions do allow for servers that send the chain out-of-order). Chain validation then tries to find a root certificate in the trust store that matches the issuer of the last certificate in that chain (furthest from the end-certificate).

This approach is problematic with cross-signed certificates, as this example shows. Most other implementation treat the intermediate CA certificates sent by the server as a pool from which to draw missing elements in the chain, and they try multiple paths until a valid one is found.

The most reliable way to resolve this is to reconfigure the server to not send the cross-signing intermediate CA once the issuing root CA is invalidated. If the CA trust store used on the Erlang/Elixir side includes the (self-signed) equivalent root CA, then verification of the shorter chain will succeed.

There is no easy way to fix this on the client side in a generic way. A workaround for specific cases should be possible by passing a ‘partial chain fun’.

3 Likes
17

Actually it seems removing the expired root CA from the CA trust store seems to work, at least for some endpoints and some OTP versions. I don’t have time to dig into this further right now, but that’s something you may want to try…

2 Likes
18

Ah, I guess that’s the ssl_verify_fun package that’s used by Hackney (and therefore HTTPoison and, depending on configuration, Tesla) doing partial chain verification. It doesn’t fix the issue for applications that rely on :ssl standard verification, including Mint.

1 Like
19

Discussion and workarounds for hackney based clients: https://github.com/benoitc/hackney/issues/633

2 Likes
20

There is a PR opened for erlang-certifi here: https://github.com/certifi/erlang-certifi/pull/39

1 Like