chulkilee

chulkilee

Strange httpc/hackeny behavior on ssl verify

I need to call HTTP API verifying SSL or not. However, I found both httpc and hackney behave strange - ssl option stay across different functions.

See this code:

bad_url1 = "https://expired.badssl.com/"
bad_url2 = "https://wrong.host.badssl.com/"

# Testing httpc

# httpc by default does not check ssl
{:ok, _} = :httpc.request(:get, {to_charlist(bad_url1), []}, [], [])

# verify ssl
{:error, _} =
  :httpc.request(
    :get,
    {to_charlist(bad_url1), []},
    [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile()]],
    []
  )

# no verify again
{:ok, _} = :httpc.request(:get, {to_charlist(bad_url1), []}, [ssl: [verify: :verify_none]], [])

# verify ssl... DOES NOT work!
{:ok._()} =
  :httpc.request(
    :get,
    {to_charlist(bad_url1), []},
    [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile()]],
    []
  )

# Testing hackney
Application.ensure_all_started(:hackney)

{:error, _} = :hackney.request(:get, bad_url1, [], [], [])
{:error, _} = :hackney.request(:get, bad_url2, [], [], [])
# => :error, as expected

{:ok, 200, _headers, ref} = :hackney.request(:get, bad_url1, [], [], [:insecure])

# before reading the body, other requests without insecure fail
{:error, _} = :hackney.request(:get, bad_url1, [], [], [])

# once the body is read..
{:ok, _body} = :hackney.body(ref)
# => :ok, as expected

{:ok, 200, _headers, ref} = :hackney.request(:get, bad_url1, [], [], [])
{:ok, _body} = :hackney.body(ref)
# WHAT? as insecure is missing, this should fail due to ssl error!

At first I thought hackney keeps some options in pool (I created insecure option for a host is persistent in a pool · Issue #570 · benoitc/hackney · GitHub ) - but apparently httpc has the same issue.

Anyone had this issue?

Marked As Solved

LostKobrakai

LostKobrakai

You might want to watch this:

The important point for your issue. Erlang will cache tls verifications.

Also Liked

voltone

voltone

In most applications, the decision whether a given TLS peer is trusted is determined by a global policy: either all is well (known issuer, not expired, correct hostname, etc.) or something is wrong, the decision does not depend on any kind of context. TLS session resumption and HTTP keep-alive are performance optimisations that work very well as long as this assumption holds.

In my talk I mentioned testing (automatic, or manual) as a scenario where these features can lead to unexpected results, and I pointed out how to disable them to get predictable results. But I do believe most users will want to leave them enabled in production.

In the scenario you describe, where some other process establishes unverified TLS connections, the real solution is of course to get that other process fixed.

If this is not possible, you’d have to take control of some of the lower layers yourself, to keep the secure and insecure parts of your application separate. At the HTTP layer you may be able to set up different connection pools (in :httpc, define a custom ‘profile’ and use it instead of the default one; the profile is the ‘reference’ you mentioned, it’s just that there is an implicit default). At the TLS layer you’d have to disable connection reuse, or manage it yourself using the client_reuse_sessions: :save and client_reuse_session options.

Mint only helps in that it does not use a connection pool, but at the TLS layer it uses the standard :ssl application, with the same connection reuse behaviour as other HTTP clients.

chulkilee

chulkilee

Thanks!

I found hackney / httpc reuses http connection across different function calls, even they do not pass same references. This is bad behavior - as your SSL option might not be honored if other process calls the “bad cert” URL with verify_none. Wow.

I really hope mint get adopted. No state MUST be shared without explicit passing references!

Last Post!

chulkilee

chulkilee

I think we’re on the same page in that 1) assumption of global policy and 2) global optimization based on the assumption leads to surprising behavior, and to make it work it needs “workaround” - not by using direct reference of connection but separating connection pool manually.

Also I understand that this is not http-level behavior/config, so http library may not need to handle this.

However, as far as a http client library supports SSL transport, it should handle this level issues “correctly”.

It is not just secure vs insecure - e.g. client cert auth will be broken if https connections are blindly shared.

Also it’s very disappointing that this assumption and behavior is not documented well in those libraries (implementing http pool connection and ssl support) - since this is important “side effect”.

For me, the best solution is to have deterministic pool name based on ssl option and pass it to “control” the behavior of http libraries (as discussed as a workaround)

Ideally, http client implementing pool connection with ssl transport layer should do that. It can use pool name with “default_#{inspect(ssloptions)}” :wink: - and need to prune unused pool nicely.

Where Next?

Popular in Questions Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 31494 112
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New

We're in Beta

About us Mission Statement