Get SSL certificate with an HTTPoison or :hackney request

Most HTTP clients abstract away the connection layer: they let you pass in the options to tune the connection establishment, but when they return the response they don’t include any connection information. Even Mint, which give you quite a lot of control over the lower layers, doesn’t let you extract the server certificate using a public API.

Having said that, it is possible to kind of make this work with HTTPoison. First of all, we’ll use ‘async’ mode, in which the response is delivered to a process mailbox. Then we’ll hook into the certificate verification callback and deliver the server cert (if it is valid) to the same mailbox in a custom message:

HTTPoison.get(
  "https://blog.voltone.net/",
  [],
  stream_to: self(),
  proxy: {"localhost", 8080},
  hackney: [
    ssl_options: [
      versions: [:"tlsv1.2"],
      ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
      cacertfile: :certifi.cacertfile(),
      depth: 3,
      verify_fun: {fn
        _, {:bad_cert, _} = reason, _ ->
          {:fail, reason}

        _,{:extension, _}, state ->
          {:unknown, state}

        _, :valid, state ->
          {:valid, state}

        peer_cert, :valid_peer, pid ->
          if :public_key.pkix_verify_hostname(peer_cert, [dns_id: 'blog.voltone.net'], match_fun: :public_key.pkix_verify_hostname_match_fun(:https)) do            
            send(pid, {:peer_cert, peer_cert})
            {:valid, pid}
          else
            {:fail, :hostname_check_failed}
          end
      end, self()}
    ]
  ]
)

Update the proxy address as needed, and remember to update the destination hostname in two places: the URI at the top, and the hostname verification function further down. Of course you could wrap this in a function that takes care of this automatically.

The result might look something like this:

iex(1)> HTTPoison.get(...)
{:ok, %HTTPoison.AsyncResponse{id: #Reference<0.1211898004.3575906308.200659>}}
iex(2)> flush
{:peer_cert,
 {:OTPCertificate,
  #...
 }
}
%HTTPoison.AsyncStatus{
  code: 200,
  id: #Reference<0.1211898004.3575906308.200659>
}
%HTTPoison.AsyncHeaders{
  headers: [
    #...
  ]
}
#...

The certificate is sent as an Erlang record. You can convert it to other formats (DER, PEM) using the :public_key module.

Finally, keep in mind that not every HTTP request necessarily results in a TLS handshake, and therefore a peer certificate message. The HTTP client may keep the connection open for reuse by multiple HTTP requests, and even if a new connection is established, TLS session resumption may cause the handshake to be skipped (pass reuse_sessions: false in the ssl_options to prevent this).

Thank you, @voltone! This is exactly the help I needed. I will also be using your excellent X509 package to process the resulting certificate.

One more boost for @voltone : couple this answer with the recursive approach in this blog post series by @alvises and you can pattern match on just the certificate and ignore the rest of the response chunks.

Well, if you don’t need the HTTP response at all you can take a few shortcuts. Drop the stream_to: self() option, ignore the response from HTTPoison.get (or check only for connection errors) and then see if you got a certificate:

receive do
  {:peer_cert, peer_cert} ->
    {:ok, peer_cert}
after
  0 ->
    {:error, "No TLS handshake, or handshake failed"}
end

You can also change the method to HEAD, to avoid transferring data you’re going to throw away anyway.

At this point I’m wondering whether you need an HTTP client at all. You could just open a TCP connection to the proxy, send "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n", receive the "HTTP/1.1 200 Connection established\r\n\r\n" reply, then upgrade the connection to TLS with :ssl.connect/2 with all the necessary ssl_options. If all goes well, call :ssl.peercert/1 to get the peer certificate.

{:ok, tcp} = :gen_tcp.connect('proxy.host.name', 8080, active: false)
:gen_tcp.send(tcp, "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n")
{:ok, "HTTP/1.1 200" <> _} = :gen_tcp.recv(tcp, 0)
{:ok, ssl} = :ssl.connect(tcp, [
  versions: [:"tlsv1.2"],
  ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
  server_name_indication: 'some.host.name',
  verify: :verify_peer,
  cacertfile: :certifi.cacertfile(),
  depth: 3,
  customize_hostname_check: [
    match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
  ]
])
{:ok, der} = :ssl.peercert(ssl)

You should now have the peer certificate in DER binary format.

Thanks for this updated approach. To pass proxy authentication credentials, would you send a Proxy-Authorization string along with the CONNECT string? Something like:

:gen_tcp.send(tcp, "CONNECT some.host.name:443 HTTP/1.1\r\nProxy-Authorization: Basic #{:base64.encode_to_string('username:password')}\r\n")

Something like that. But you’re missing the empty line at the end of the request there: you need another \r\n.

Thanks, again, @voltone.. I really appreciate your help with this.