Strange httpc/hackeny behavior on ssl verify

Questions / Help
, , ,
1

I need to call HTTP API verifying SSL or not. However, I found both httpc and hackney behave strange - ssl option stay across different functions.

See this code:

bad_url1 = "https://expired.badssl.com/"
bad_url2 = "https://wrong.host.badssl.com/"

# Testing httpc

# httpc by default does not check ssl
{:ok, _} = :httpc.request(:get, {to_charlist(bad_url1), []}, [], [])

# verify ssl
{:error, _} =
  :httpc.request(
    :get,
    {to_charlist(bad_url1), []},
    [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile()]],
    []
  )

# no verify again
{:ok, _} = :httpc.request(:get, {to_charlist(bad_url1), []}, [ssl: [verify: :verify_none]], [])

# verify ssl... DOES NOT work!
{:ok._()} =
  :httpc.request(
    :get,
    {to_charlist(bad_url1), []},
    [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile()]],
    []
  )

# Testing hackney
Application.ensure_all_started(:hackney)

{:error, _} = :hackney.request(:get, bad_url1, [], [], [])
{:error, _} = :hackney.request(:get, bad_url2, [], [], [])
# => :error, as expected

{:ok, 200, _headers, ref} = :hackney.request(:get, bad_url1, [], [], [:insecure])

# before reading the body, other requests without insecure fail
{:error, _} = :hackney.request(:get, bad_url1, [], [], [])

# once the body is read..
{:ok, _body} = :hackney.body(ref)
# => :ok, as expected

{:ok, 200, _headers, ref} = :hackney.request(:get, bad_url1, [], [], [])
{:ok, _body} = :hackney.body(ref)
# WHAT? as insecure is missing, this should fail due to ssl error!

At first I thought hackney keeps some options in pool (I created https://github.com/benoitc/hackney/issues/570 ) - but apparently httpc has the same issue.

Anyone had this issue?

2

You might want to watch this:

The important point for your issue. Erlang will cache tls verifications.

3 Likes
3

Thanks!

I found hackney / httpc reuses http connection across different function calls, even they do not pass same references. This is bad behavior - as your SSL option might not be honored if other process calls the “bad cert” URL with verify_none. Wow.

I really hope mint get adopted. No state MUST be shared without explicit passing references!

3 Likes
4

In most applications, the decision whether a given TLS peer is trusted is determined by a global policy: either all is well (known issuer, not expired, correct hostname, etc.) or something is wrong, the decision does not depend on any kind of context. TLS session resumption and HTTP keep-alive are performance optimisations that work very well as long as this assumption holds.

In my talk I mentioned testing (automatic, or manual) as a scenario where these features can lead to unexpected results, and I pointed out how to disable them to get predictable results. But I do believe most users will want to leave them enabled in production.

In the scenario you describe, where some other process establishes unverified TLS connections, the real solution is of course to get that other process fixed.

If this is not possible, you’d have to take control of some of the lower layers yourself, to keep the secure and insecure parts of your application separate. At the HTTP layer you may be able to set up different connection pools (in :httpc, define a custom ‘profile’ and use it instead of the default one; the profile is the ‘reference’ you mentioned, it’s just that there is an implicit default). At the TLS layer you’d have to disable connection reuse, or manage it yourself using the client_reuse_sessions: :save and client_reuse_session options.

Mint only helps in that it does not use a connection pool, but at the TLS layer it uses the standard :ssl application, with the same connection reuse behaviour as other HTTP clients.

4 Likes
5

I think we’re on the same page in that 1) assumption of global policy and 2) global optimization based on the assumption leads to surprising behavior, and to make it work it needs “workaround” - not by using direct reference of connection but separating connection pool manually.

Also I understand that this is not http-level behavior/config, so http library may not need to handle this.

However, as far as a http client library supports SSL transport, it should handle this level issues “correctly”.

It is not just secure vs insecure - e.g. client cert auth will be broken if https connections are blindly shared.

Also it’s very disappointing that this assumption and behavior is not documented well in those libraries (implementing http pool connection and ssl support) - since this is important “side effect”.

For me, the best solution is to have deterministic pool name based on ssl option and pass it to “control” the behavior of http libraries (as discussed as a workaround)

Ideally, http client implementing pool connection with ssl transport layer should do that. It can use pool name with “default_#{inspect(ssloptions)}” :wink: - and need to prune unused pool nicely.