TLS handshake failure running mix local.hex

Questions / Help
1

Starting to follow the phoenix beta book is taking me a bit further than expected…

When running “mix local.hex” I get the following error:

ps@x301:~> export MIX_DEBUG=1; mix local.hex
** Running mix loadconfig
** Running mix local.hex

09:59:18.151 [info]  ['TLS', 32, 'client', 58, 32, 73, 110, 32, 115, 116, 97, 116, 101, 32, 'hello', 32, 'received SERVER ALERT: Fatal - Handshake Failure', 10]

09:59:18.208 [info]  ['TLS', 32, 'client', 58, 32, 73, 110, 32, 115, 116, 97, 116, 101, 32, 'hello', 32, 'received SERVER ALERT: Fatal - Handshake Failure', 10]
** (Mix.Error) httpc request failed with: {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet6, [:inet6], {:tls_alert, 'handshake failure'}}, {:inet, [:inet], {:tls_alert, 'handshake failure'}}]}

Could not install Hex because Mix could not download metadata at http://repo.hex.pm/installs/hex-1.x.csv.

    (mix) lib/mix.ex:323: Mix.raise/1
    (mix) lib/mix/local.ex:126: Mix.Local.find_matching_versions_from_signed_csv!/2
    (mix) lib/mix/tasks/local.hex.ex:56: Mix.Tasks.Local.Hex.run_install/1
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2

This is on opensuse tumbleweed.

ps@x301:~> elixir --version
Erlang/OTP 21 [erts-10.1.3] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]

Elixir 1.7.4 (compiled with Erlang/OTP 21)

Information for package erlang:

Repository : Erlang
Name : erlang
Version : 21.1.4-1.2
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:languages:erlang

Following in iex gives me the same error. I would like to play more with this but can’t figure out how to give ssl options and the likes.
:inets.start
:ssl.start
:httpc.request ‘https://repo.hex.pm’

Anybody any idea what is wrong here or how I can troubleshoot further?

thanks!

2

What’s the full result of curl -LI http://repo.hex.pm/installs/hex-1.x.csv in your shell first of all?

3

ps@x301:~> curl -LI http://repo.hex.pm/installs/hex-1.x.csv
HTTP/1.1 301 Moved Permanently
Server: Varnish
Retry-After: 0
Content-Type:
Content-Length: 0
Accept-Ranges: bytes
Date: Tue, 11 Dec 2018 17:16:00 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-ams21050-AMS
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1544548561.798352,VS0,VE2
Location: https://repo.hex.pm/installs/hex-1.x.csv

HTTP/2 200
x-amz-id-2: nz0V9reAyWhZYmTJJ3IdCn9OiSoUKeAnAFBeYWGQEQozUHNTAo3maw/EcwrWh8dVwsyj6A7ojp8=
x-amz-request-id: 307AFAFF3A1DB7F4
last-modified: Thu, 08 Nov 2018 07:57:51 GMT
etag: “22dd7562e6a54f93910c4300a48e1878”
cache-control: public, max-age=604800
x-amz-meta-surrogate-key: installs
x-amz-version-id: tsBMBMdBUUnNEnDX2R6F5jlVB3jj.lcY
content-type: text/csv
server: AmazonS3
accept-ranges: bytes
date: Tue, 11 Dec 2018 17:16:00 GMT
via: 1.1 varnish
age: 570868
x-served-by: cache-iad2134-IAD, cache-ams21020-AMS
x-cache: HIT, HIT
x-cache-hits: 4422, 545
x-timer: S1544548561.920767,VS0,VE0
content-length: 14626

Thanks!

4

Cool, so same as me and thus no network issues. I wonder if this is the tcp6 issues that some people experience with the BEAM. I don’t know how to force hex to use tcp though, might need to wait for one of the maintainers to show up.

╰─➤  digall repo.hex.pm                                                                                                           1 ↵
repo.hex.pm.            3788 IN HINFO "ANY obsoleted" "See draft-ietf-dnsop-refuse-any"
repo.hex.pm.            3788 IN RRSIG HINFO 13 3 3789 (
                                20181212190644 20181210170644 34505 hex.pm.
                                qTweB/lpWyy6lWcEPs1mLqMeHJ3VfMWFJoBJ/dCnjAmj
                                8KlEhVel1wIhWe2S/oR8HLkNDxZZkjnVEHlcCs6d+Q== )

The heck is HINFO and RRSIG… o.O
Interesting…, regardless:

repo.hex.pm.            299     IN      CNAME   dualstack.f2.shared.global.fastly.net.
dualstack.f2.shared.global.fastly.net. 29 IN A  151.101.2.2
dualstack.f2.shared.global.fastly.net. 29 IN A  151.101.66.2
dualstack.f2.shared.global.fastly.net. 29 IN A  151.101.130.2
dualstack.f2.shared.global.fastly.net. 29 IN A  151.101.194.2

Not seeing any tcpv6 addresses anyway, so why is :inet6 being used in your log…

While waiting for others, perhaps you can update to Erlang 21.2 that came out today? I think it touched the SSL system so maybe it has a fix for your setup?

5

Does below (from my original post) not mean it tries both ipv6 and ipv4? I have both ipv4 and ipv6 here.

So it refuses type “ANY” queries, which maybe is exactly what mix, hex, httpc is asking for. That would mean if it fails on ipv6 it does not try on ipv4.
When I dig for AAAA specifically I do get an ipv6 address.

ps@x301:~> dig -t AAAA repo.hex.pm
…

;; QUESTION SECTION:
;repo.hex.pm. IN AAAA

;; ANSWER SECTION:
repo.hex.pm. 260 IN CNAME dualstack.f2.shared.global.fastly.net.
dualstack.f2.shared.global.fastly.net. 10 IN AAAA 2a04:4e42:9::514

I don’t see Erlang 21.2 yet in the tumbleweed repositories or on the build site. I guess I will wait a little longer. It feels like an erlang ssl problem or perhaps a DNS issue.
I also set up a Debian virtual machine. There this did work but very erratically. Using a mirror solved things there, but it remains slow and works only sometimes.(that may be just a feeling as I did quite some troubleshooting)
Just starting the book, it doesn’t give me a warm feeling.

6

Oh wow, it does! That’s why my digall command returned squat-all! How’d that happen?! o.O

I get this for an AAAA test:

repo.hex.pm.            299     IN      CNAME   dualstack.f2.shared.global.fastly.net.
dualstack.f2.shared.global.fastly.net. 29 IN AAAA 2a04:4e42::514
dualstack.f2.shared.global.fastly.net. 29 IN AAAA 2a04:4e42:200::514
dualstack.f2.shared.global.fastly.net. 29 IN AAAA 2a04:4e42:400::514
dualstack.f2.shared.global.fastly.net. 29 IN AAAA 2a04:4e42:600::514

Which appears distinctly different than yours of 2a04:4e42:9::514… interesting…

Maybe set the IP in your hosts file directly of repo.hex.pm as a test to bypass the DNS?

7

Putting that ip address in /etc/hosts does not help. Exact same error.
Back to the erlang ssl module? I do not know how to troubleshoot this.
Can you or anyone give me an example of how to pass protocol versions, ciphers and the likes to either :ssl.connect or :httpc.request? I have searched but fail to translate the erlang syntax into elixir. (as I say, I’m only at the beginning of the book :wink: )

8

You can pass ssl options like this: :httpc.request(:get, {'https://repo.hex.pm', []}, [ssl: [versions: [:"tlsv1.2"]]], [])

It is possible to trace the TLS handshake messages inside the ssl application:

:dbg.tracer()
:dbg.p(:all, [:call])
:dbg.tpl(:tls_handshake, :get_tls_handshake, :x)

The output shows both the TLS frames exchanged with the server and the client-side state. We can compare that with a successful connection and see if anything is different.

4 Likes
9

The problem is the follow:

httpc:request(get, {"https://repo.hex.pm", []}, [], []).
=INFO REPORT==== 11-Dec-2018::23:05:11.208204 ===
TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure

{error,{failed_connect,[{to_address,{"repo.hex.pm",443}},
                        {inet,[inet],{tls_alert,"handshake failure"}}]}}

by using all the ciphers it works:

All = ssl:cipher_suites(all, 'tlsv1.2').
Opts = [{ciphers, All}].
 httpc:request(get, {"https://repo.hex.pm", []}, [{ssl,Opts}], []).

I will investigate the package:

3 Likes
10

Thanks, this is very helpful but didn’t get me more information unfortunately.
It gives me the exact same error, no debugging output, which makes me think it never even gets to that handshake. (On a Debian virtual machine this yields a ton of output.)

:httpc.request(:get, {‘https://repo.hex.pm’, }, [ssl: [versions: [:“tlsv1.2”]]], )

09:23:12.721 [info] [‘TLS’, 32, ‘client’, 58, 32, 73, 110, 32, 115, 116, 97, 116, 101, 32, ‘hello’, 32, ‘received SERVER ALERT: Fatal - Handshake Failure’, 10]
{:error,
{:failed_connect,
[
{:to_address, {‘repo.hex.pm’, 443}},
{:inet, [:inet], {:tls_alert, ‘handshake failure’}}
]}}

I tried this with several websites, all give the same error.
Voltone, do you know if there are other :dbg calls that might make me wiser?

Thanks for the help all!
I contacted a maintainer of the opensuse tumbleweed Erlang package also. He feels it is a tls issue.

2 Likes
11

Hmm, so maybe the combination of the patch, the OTP 21 defaults (no more RSA Kx by default) and an old OpenSSL version results in a very short list of default cipher suites that causes interop issues with some servers. That might explain the issue with raw httpc.

What’s the output of ssl:cipher_suites(default, 'tlsv1.2')?

Still, I thought hex was using a specific set of cipher suites that includes RSA Kx:

So not sure it would explain everything…

1 Like
12

Ah, mix local.hex is not part of Hex, so I guess it doesn’t use that cipher list.

1 Like
13

For me this is:

iex(18)> :ssl.cipher_suites(:default, :“tlsv1.2”)
[
%{cipher: :aes_256_gcm, key_exchange: :ecdhe_ecdsa, mac: :aead, prf: :sha384},
%{cipher: :aes_256_gcm, key_exchange: :ecdhe_rsa, mac: :aead, prf: :sha384},
%{
cipher: :aes_256_cbc,
key_exchange: :ecdhe_ecdsa,
mac: :sha384,
prf: :sha384
},
%{cipher: :aes_256_cbc, key_exchange: :ecdhe_rsa, mac: :sha384, prf: :sha384},
%{cipher: :aes_256_gcm, key_exchange: :ecdh_ecdsa, mac: :aead, prf: :sha384},
%{cipher: :aes_256_gcm, key_exchange: :ecdh_rsa, mac: :aead, prf: :sha384},
%{cipher: :aes_256_cbc, key_exchange: :ecdh_ecdsa, mac: :sha384, prf: :sha384},
%{cipher: :aes_256_cbc, key_exchange: :ecdh_rsa, mac: :sha384, prf: :sha384},
%{cipher: :aes_256_gcm, key_exchange: :dhe_rsa, mac: :aead, prf: :sha384},
%{cipher: :aes_256_gcm, key_exchange: :dhe_dss, mac: :aead, prf: :sha384},
%{
cipher: :aes_256_cbc,
key_exchange: :dhe_rsa,
mac: :sha256,
prf: :default_prf
},
%{
cipher: :aes_256_cbc,
key_exchange: :dhe_dss,
mac: :sha256,
prf: :default_prf
},
%{cipher: :aes_128_gcm, key_exchange: :ecdhe_ecdsa, mac: :aead, prf: :sha256},
%{cipher: :aes_128_gcm, key_exchange: :ecdhe_rsa, mac: :aead, prf: :sha256},
%{
cipher: :aes_128_cbc,
key_exchange: :ecdhe_ecdsa,
mac: :sha256,
prf: :sha256
},
%{cipher: :aes_128_cbc, key_exchange: :ecdhe_rsa, mac: :sha256, prf: :sha256},
%{cipher: :aes_128_gcm, key_exchange: :ecdh_ecdsa, mac: :aead, prf: :sha256},
%{cipher: :aes_128_gcm, key_exchange: :ecdh_rsa, mac: :aead, prf: :sha256},
%{cipher: :aes_128_cbc, key_exchange: :ecdh_ecdsa, mac: :sha256, prf: :sha256},
%{cipher: :aes_128_cbc, key_exchange: :ecdh_rsa, mac: :sha256, prf: :sha256},
%{cipher: :aes_128_gcm, key_exchange: :dhe_rsa, mac: :aead, prf: :sha256},
%{cipher: :aes_128_gcm, key_exchange: :dhe_dss, mac: :aead, prf: :sha256},
%{
cipher: :aes_128_cbc,
key_exchange: :dhe_rsa,
mac: :sha256,
prf: :default_prf
},
%{
cipher: :aes_128_cbc,
key_exchange: :dhe_dss,
mac: :sha256,
prf: :default_prf
},
%{
cipher: :aes_256_cbc,
key_exchange: :ecdhe_ecdsa,
mac: :sha,
prf: :default_prf
},
%{
cipher: :aes_256_cbc,
key_exchange: :ecdhe_rsa,
mac: :sha,
prf: :default_prf
},
%{cipher: :aes_256_cbc, key_exchange: :dhe_rsa, mac: :sha, prf: :default_prf},
%{cipher: :aes_256_cbc, key_exchange: :dhe_dss, mac: :sha, prf: :default_prf},
%{
cipher: :aes_256_cbc,
key_exchange: :ecdh_ecdsa,
mac: :sha,
prf: :default_prf
},
%{cipher: :aes_256_cbc, key_exchange: :ecdh_rsa, mac: :sha, prf: :default_prf},
%{
cipher: :aes_128_cbc,
key_exchange: :ecdhe_ecdsa,
mac: :sha,
prf: :default_prf
},
%{
cipher: :aes_128_cbc,
key_exchange: :ecdhe_rsa,
mac: :sha,
prf: :default_prf
},
%{cipher: :aes_128_cbc, key_exchange: :dhe_rsa, mac: :sha, prf: :default_prf},
%{cipher: :aes_128_cbc, key_exchange: :dhe_dss, mac: :sha, prf: :default_prf},
%{
cipher: :aes_128_cbc,
key_exchange: :ecdh_ecdsa,
mac: :sha,
prf: :default_prf
},
%{cipher: :aes_128_cbc, key_exchange: :ecdh_rsa, mac: :sha, prf: :default_prf}
]

14

That looks pretty normal, I don’t see why the Hex server (or rather, Fastly’s CDN) wouldn’t find a suitable cipher in this list. Does @Gsantomaggio’s workaround with :ssl.cipher_suites(:all, :"tlsv1.2") work for you?

15

Yes, I also had the impression it is quite an extensive array of ciphers available.
How do I translate this:

All = ssl:cipher_suites(all, 'tlsv1.2').
Opts = [{ciphers, All}].
 httpc:request(get, {"https://repo.hex.pm", []}, [{ssl,Opts}], []).

to Elixir? The first command already fails:

iex(25)> All=:ssl.cipher_suites(:all, :“tlsv1.2”)
** (MatchError) no match of right hand side value: [%{cipher: :aes_256_gcm, key_exchange: :ecdhe_ecdsa, mac: :aead, prf: :sha384}, %{cipher: :aes_256_gcm, key_exchange: :ecdhe_rsa, mac: :aea

In erl it does seem to make sense but the httpc request yields:

httpc:request(get, {“https://repo.hex.pm”, }, [{ssl,Opts}], ).
** exception exit: {noproc,
{gen_server,call,
[httpc_manager,
{request,
{request,undefined,<0.97.0>,0,https,
{“repo.hex.pm”,443},
“/”,,get,
{http_request_h,undefined,“keep-alive”,undefined,undefined,
undefined,undefined,undefined,undefined,undefined,…},
{,},
{http_options,“HTTP/1.1”,infinity,true,
{essl,[{…}]},
undefined,false,infinity,…},
“https://repo.hex.pm”,,none,,1544618685164,undefined,
undefined,undefined,…}},
infinity]}}
in function gen_server:call/3 (gen_server.erl, line 223)
in call from httpc:handle_request/9 (httpc.erl, line 565)

1 Like
16

In the meantime, if you just want to get past the mix local.hex stage, you should be able to install Hex manually:

mkdir -p ~/.mix/archives
cd ~/.mix/archives
wget https://repo.hex.pm/installs/1.7.0/hex-0.18.2.ez
unzip hex-0.18.2.ez

(I think)

Edit note the fixed typo

1 Like
17 
:ssl.start()
:inets.start()
all = :ssl.cipher_suites(:all, :"tlsv1.2")
opts = [ciphers: all]
:httpc.request(:get, {'https://repo.hex.pm', []}, [ssl: opts], [])
1 Like
18

Thanks, I found something along those lines also, but gave up when I noticed that several other tasks failed also when having to pull anything from the net. I may try it again until problems get a more definite solution.

19

Broke out of iex and forgot to start ssl again, it is working now indeed as Gsantomaggio says.
Now, if I start mix as iex -S mix local.hex could I somehow set the ssl parameters? Just guessing here to find a more acceptable temporary solution.

The repo.hex.pm certificate matches with the cipher:

#{cipher => aes_128_gcm,key_exchange => ecdhe_rsa,mac => aead, prf => sha256}

This cipher is however available in :default / tlsv1.2
I do see that the certificate has a large amount of Certificate Subject Alt Names, 130 in all, many wildcard domains.

Enough for now, I will wait for news from Gsantomaggio.

20

No, none that I can think of.

I found some time to bring up a VM with a similar environment (I get a slightly different Erlang version, but close enough), and I was able to reproduce the problem. The TLS handshake shows the client offering a couple of cipher suites with DHE key exchange, and I guess the repo endpoints at Fastly do not support that for performance reasons.

They probably support ECDHE and RSA key exchange. The former is not supported by the Erlang installation (if I run :ssl.eccs() I get , which mean no ECC curves are supported), and the latter is disabled in OTP >=21 because of security concerns. When explicitly enabling all cipher suites, directly with httpc, the RSA suites are enabled and the handshake succeeds.

I’ll look into it some more and see if I can think of a good workaround…

1 Like