toddholmberg

toddholmberg

Get SSL certificate with an HTTPoison or :hackney request

Hello. I am trying to find the best way forward doing the following:

  1. Make an HTTPS request to an arbitrary URL.
  2. Make this request using a proxy.
  3. Be able to set the TLS/SSL protocol used for this connection.
  4. Be able to set the protocol-appropriate cipher used for this connection.
  5. Return the associated SSL certificate.

HTTPoison allows setting SSL options via its hackney foundation, but does not have return the SSL certificate with the response.

The Erlang :ssl library will allow setting the protocol and cipher for a request, but I am having trouble navigating the specifics of how to do this request via a proxy.

Has anyone had any experience making such a request and getting the associated SSL certificate? Thanks in advance for any advice.

Marked As Solved

voltone

voltone

Most HTTP clients abstract away the connection layer: they let you pass in the options to tune the connection establishment, but when they return the response they don’t include any connection information. Even Mint, which give you quite a lot of control over the lower layers, doesn’t let you extract the server certificate using a public API.

Having said that, it is possible to kind of make this work with HTTPoison. First of all, we’ll use ‘async’ mode, in which the response is delivered to a process mailbox. Then we’ll hook into the certificate verification callback and deliver the server cert (if it is valid) to the same mailbox in a custom message:

HTTPoison.get(
  "https://blog.voltone.net/",
  [],
  stream_to: self(),
  proxy: {"localhost", 8080},
  hackney: [
    ssl_options: [
      versions: [:"tlsv1.2"],
      ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
      cacertfile: :certifi.cacertfile(),
      depth: 3,
      verify_fun: {fn
        _, {:bad_cert, _} = reason, _ ->
          {:fail, reason}

        _,{:extension, _}, state ->
          {:unknown, state}

        _, :valid, state ->
          {:valid, state}

        peer_cert, :valid_peer, pid ->
          if :public_key.pkix_verify_hostname(peer_cert, [dns_id: 'blog.voltone.net'], match_fun: :public_key.pkix_verify_hostname_match_fun(:https)) do            
            send(pid, {:peer_cert, peer_cert})
            {:valid, pid}
          else
            {:fail, :hostname_check_failed}
          end
      end, self()}
    ]
  ]
)

Update the proxy address as needed, and remember to update the destination hostname in two places: the URI at the top, and the hostname verification function further down. Of course you could wrap this in a function that takes care of this automatically.

The result might look something like this:

iex(1)> HTTPoison.get(...)
{:ok, %HTTPoison.AsyncResponse{id: #Reference<0.1211898004.3575906308.200659>}}
iex(2)> flush
{:peer_cert,
 {:OTPCertificate,
  #...
 }
}
%HTTPoison.AsyncStatus{
  code: 200,
  id: #Reference<0.1211898004.3575906308.200659>
}
%HTTPoison.AsyncHeaders{
  headers: [
    #...
  ]
}
#...

The certificate is sent as an Erlang record. You can convert it to other formats (DER, PEM) using the :public_key module.

Finally, keep in mind that not every HTTP request necessarily results in a TLS handshake, and therefore a peer certificate message. The HTTP client may keep the connection open for reuse by multiple HTTP requests, and even if a new connection is established, TLS session resumption may cause the handshake to be skipped (pass reuse_sessions: false in the ssl_options to prevent this).

Also Liked

voltone

voltone

Well, if you don’t need the HTTP response at all you can take a few shortcuts. Drop the stream_to: self() option, ignore the response from HTTPoison.get (or check only for connection errors) and then see if you got a certificate:

receive do
  {:peer_cert, peer_cert} ->
    {:ok, peer_cert}
after
  0 ->
    {:error, "No TLS handshake, or handshake failed"}
end

You can also change the method to HEAD, to avoid transferring data you’re going to throw away anyway.

At this point I’m wondering whether you need an HTTP client at all. You could just open a TCP connection to the proxy, send "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n", receive the "HTTP/1.1 200 Connection established\r\n\r\n" reply, then upgrade the connection to TLS with :ssl.connect/2 with all the necessary ssl_options. If all goes well, call :ssl.peercert/1 to get the peer certificate.

{:ok, tcp} = :gen_tcp.connect('proxy.host.name', 8080, active: false)
:gen_tcp.send(tcp, "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n")
{:ok, "HTTP/1.1 200" <> _} = :gen_tcp.recv(tcp, 0)
{:ok, ssl} = :ssl.connect(tcp, [
  versions: [:"tlsv1.2"],
  ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
  server_name_indication: 'some.host.name',
  verify: :verify_peer,
  cacertfile: :certifi.cacertfile(),
  depth: 3,
  customize_hostname_check: [
    match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
  ]
])
{:ok, der} = :ssl.peercert(ssl)

You should now have the peer certificate in DER binary format.

toddholmberg

toddholmberg

Thank you, @voltone! This is exactly the help I needed. I will also be using your excellent X509 package to process the resulting certificate.

toddholmberg

toddholmberg

One more boost for @voltone : couple this answer with the recursive approach in this blog post series by @alvises and you can pattern match on just the certificate and ignore the rest of the response chunks.

Last Post!

toddholmberg

toddholmberg

Thanks, again, @voltone.. I really appreciate your help with this.

Where Next?

Popular in Questions Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44532 311
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44139 214
New

We're in Beta

About us Mission Statement