HTTPS handshake error: Fatal - Handshake Failure

I’m having the same problem. This is for code that was working previously with the same site. Although I have no idea if the site has made changes to their server, like updating their allowed encryption or something.

The server aborts the handshake, so it looks like it doesn’t like something the client sends. Unfortunately the server does not (cannot) indicate exactly what it didn’t like through the TLS alert mechanism. It might record more details in its local log, but I’m guessing you don’t have access to that log.

I would trace a successful handshake (using curl or openssl) and an unsuccessful one with Wireshark, and compare. You can also trace the handshake by adding log_level: :debug to the ssl options in the client, but that might be hard to compare with the output of other clients.

You can encounter this error for multiple reasons, one of them already mentioned above (conflicting TLS/cipher suite versions). But it can also be that you’re connecting HTTPS over a HTTP port. Another one is a broken erlang/BEAM install with openssl.

I would try to make really sure you’re connecting to a HTTPS(TLS) port by curling to the endpoint. If curl acts ‘normal’ you’re more sure the issue is inside your own code/side. curl -vvv https://www.example.com

You could list the supported TLS versions and cipher suites with for example SSL Server Test (Powered by Qualys SSL Labs) (if it is open to the outside world) or otherwise with nmap or openssl s_client. (nmap = nmap --script ssl-enum-ciphers -p 443 www.example.com) see here for more info how to do this.

If you can share the given endpoint and it’s open for the public I can also take a look if you want.

One final last note; maybe you already know but don’t use verify_none in production settings, it’s saying you don’t care where the server TLS certificate comes from (as in: you don’t mind it’s self signed, most of the time not what you want).

Unfortunately this is the default value for :httpc and a lot of HTTP libraries build on top of it directly or indirectly, and some of them may use this insecure default, looking at you Tesla:

https://github.com/teamon/tesla/issues/293

Now, even when HTTP libraries don’t use the insecure defaults you may “accidentally override” it when customizing the library. I say accidentally because if the library doesn’t merge the configuration you pass in with the one it uses internally to secure :httpc, then you just have replaced accidentally them.

You may want to take a look to this talk from @voltone:

Thanks for the reply. I’m able to use HTTPoison to connect to https://www.google.com as well as another personal https site I have. So I don’t suspect the installation is the culprit. I’m using HTTPS URLs and the responses from the servers indicate that the client is going to the right port.

Here’s the nmap dump of ciphers to the problem service. That list seems kind of skimpy, but I’ve only looked at a few other sites for comparison:

% nmap --script ssl-enum-ciphers -p 443 api.etrade.com                                                                                                         14:30:49
Starting Nmap 7.91 ( https://nmap.org ) at 2021-01-20 14:34 EST
Nmap scan report for api.etrade.com (12.153.224.50)
Host is up (0.027s latency).
Other addresses for api.etrade.com (not scanned): 198.93.34.32
rDNS record for 12.153.224.50: mtrader.etrade.com

PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Forward Secrecy not supported by any cipher
|_  least strength: A

Here is a naive attempt to connect to the api URL:

iex(3)> HTTPoison.get!("https://api.etrade.com")
[info] TLS :client: In state :hello received SERVER ALERT: Fatal - Handshake Failure

** (HTTPoison.Error) {:tls_alert, {:handshake_failure, 'TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure\n'}}
    (httpoison 1.8.0) lib/httpoison.ex:156: HTTPoison.request!/5

Adding a tlsv1.2 option doesn’t seem to change anything. Neither does the suggested log_level: :debug. The output was exactly the same.

iex(4)> HTTPoison.get!("https://api.etrade.com", [], [ssl: [versions: [:'tlsv1.2']], log_level: :debug])

I’m using the 1.8 version of HTTPoison. How do I see what ciphers it (or Hackney) has available?

Followup edit: I think I see the ciphers offered up by the client in the wireshark capture I just did. HTTPoison or Hackney or whatever offers no TLS_RSA_WITH_AES* cipher suites. Meanwhile, a packet capture of a successful curl negotiation includes plenty that match those available on the server.

What can I do about it?

Thank you for any advice.

HTTPoison.get!("https://api.etrade.com", [], [ssl: [ciphers: :ssl.cipher_suites() ++ [{:rsa, :aes_256_cbc, :sha256}]], log_level: :debug])

seems to work here! See if that works for you.

Excellent!

My code is working again. That’s a big relief. At first when the etrade API stopped working, I was worried that they had changed something fundamental in their OAUTH implementation that was going to involve some painstaking debugging. I’m having enough trouble finding time for this side project. I was dreading the thought at having to spend precious hours reworking code that had been working fine.

I guess that either the etrade admins stopped supporting some cipher I was using before or the erlang ssl stopped including some suites by default. Either way, your suggestion did the trick.

Thank you very much.

It seems they’ve documented it here: Erlang -- ssl

• For security reasons RSA key exchange cipher suites are no longer supported by default, but can be configured. (OTP 21)

Glad to see your code is working again!

Ah, of course, I had forgotten all about the removal of RSA key exchange already. One would expect a site such as this to support (EC)DHE: I mean, it is 2021…

Anyway, keep in mind that :ssl.cipher_suites/1,2 is deprecated. The ‘official’ way to select custom ciphers would be something like:

defaults = :ssl.cipher_suites(:default, :"tlsv1.2")
rsa_kx =
  :ssl.cipher_suites(:all, :"tlsv1.2")
  |> :ssl.filter_cipher_suites(
    key_exchange: &(&1 == :rsa),
    cipher: &(&1 in [:aes_128_cbc, :aes_128_gcm, :aes_256_cbc, :aes_256_gcm]),
  )
HTTPoison.get!("https://api.etrade.com", [], ssl: [ciphers: defaults ++ rsa_kx])

Also, and more importantly, as @Exadra37 pointed out, passing custom ssl options to Hackney/HTTPoison will override its secure defaults, including verify: :verify_peer and the CA trust store. So actually, for a secure connection you’d have to call:

HTTPoison.get!("https://api.etrade.com", [], ssl: [
  ciphers: defaults ++ rsa_kx,
  verify: :verify_peer,
  cacertfile: :certifi.cacertfile(),
  depth: 3,
  customize_hostname_check: [
    match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
  ]
])

just to add as a helper to debug these calls: :hackney_trace.enable(:max, :io) logs your requests to STDOUT on the REPL/console (it shows the enabled cipers for example).