:httpc redirect issue

tangui

Hi all,

Not an Elixir problem per se but one with Erlang’s httpc module, that is getting me insane. I’m using it in one of my projects trying to avoid adding a dependency for a few http requests.

I’m trying to get a payload at an address, and receive a redirect ({:redirect, false} added for demonstration purpose):

iex(67)> :httpc.request(:get, {'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL?token=[access_token_here]', []}, [{:verbose, :debug}, {:autoredirect, false}], [])

17:28:12.607 [info]  [73, 110, 118, 97, 108, 105, 100, 32, 111, 112, 116, 105, 111, 110, 32, [123, ['verbose', 44, 'debug'], 125], 32, 105, 103, 110, 111, 114, 101, 100, 32, 10]
{:ok,
 {{'HTTP/1.1', 301, 'MOVED PERMANENTLY'},
  [
    {'connection', 'keep-alive'},
    {'date', 'Fri, 11 Jan 2019 14:28:13 GMT'},
    {'location',
     'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=[access_token_here]'},
    {'server', 'cloudflare'},
    {'content-length', '441'},
    {'content-type', 'text/html; charset=utf-8'},
    {'set-cookie',
     '__cfduid=d88330cf7c6165a3dad6797c3cf5e8c171547216892; expires=Sat, 11-Jan-20 14:28:12 GMT; path=/; domain=.fidoalliance.org; HttpOnly'},
    {'x-frame-options', 'SAMEORIGIN'},
    {'x-xss-protection', '1; mode=block'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'cf-ray', '4978118b484e8ed9-DME'}
  ],
  '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>Redirecting...</title>\n<h1>Redirecting...</h1>\n<p>You should be redirected automatically to target URL: <a href="https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=...">https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=d...</a>.  If not click the link.'}}

Let’s try to make a new request from the value of the Location response header (the only difference is the '/' added before the '?'):

iex(68)> :httpc.request(:get, {'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=[access_token_here]', []}, [{:verbose, :debug}, {:autoredirect, false}], [])

17:30:19.542 [info]  [73, 110, 118, 97, 108, 105, 100, 32, 111, 112, 116, 105, 111, 110, 32, [123, ['verbose', 44, 'debug'], 125], 32, 105, 103, 110, 111, 114, 101, 100, 32, 10]
{:ok,
 {{'HTTP/1.1', 200, 'OK'},
  [
    {'cache-control', 'public, max-age=43200'},
    {'connection', 'keep-alive'},
    {'date', 'Fri, 11 Jan 2019 14:30:20 GMT'},
    {'accept-ranges', 'bytes'},
    {'etag', '"1547064685.0-7052-1005851543"'},
    {'server', 'cloudflare'},
    {'content-length', '7052'},
    {'content-type', 'application/octet-stream'},
    {'expires', 'Sat, 12 Jan 2019 02:30:20 GMT'},
    {'last-modified', 'Wed, 09 Jan 2019 20:11:25 GMT'},
    {'set-cookie',
     '__cfduid=d7e19909933a681371cf8aa0a1c490d191547217019; expires=Sat, 11-Jan-20 14:30:19 GMT; path=/; domain=.fidoalliance.org; HttpOnly'},
    {'x-frame-options', 'SAMEORIGIN'},
    {'content-disposition', 'attachment; filename=t3fyXjAh2Eu7EDsN4tQ6HL'},
    {'x-xss-protection', '1; mode=block'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'cf-ray', '497814a48a104e54-DME'}
  ],
  'eyJhc3NlcnRpb25TY2hlbWUiOiAiVTJGVjFCSU4iLCAiYXR0YW[...]' ++ ...}}

Works fine. But now, when I reenable the auto-redirect (by not setting the :autoredirect option - default is true), I get a 404 error:

iex(69)> :httpc.request(:get, {'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL?token=...', []}, [{:verbose, :debug}], [])                         
17:33:29.843 [info]  [73, 110, 118, 97, 108, 105, 100, 32, 111, 112, 116, 105, 111, 110, 32, [123, ['verbose', 44, 'debug'], 125], 32, 105, 103, 110, 111, 114, 101, 100, 32, 10]
{:ok,
 {{'HTTP/1.1', 404, 'NOT FOUND'},
  [
    {'connection', 'keep-alive'},
    {'date', 'Fri, 11 Jan 2019 14:33:31 GMT'},
    {'server', 'cloudflare'},
    {'content-length', '233'},
    {'content-type', 'text/html'},
    {'set-cookie',
     '__cfduid=db7e100503b6a9c7a64fcdb6f851aca681547217210; expires=Sat, 11-Jan-20 14:33:30 GMT; path=/; domain=.fidoalliance.org; HttpOnly'},
    {'x-frame-options', 'SAMEORIGIN'},
    {'x-xss-protection', '1; mode=block'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'cf-ray', '4978194eebbf8eeb-DME'}
  ],
  '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n'}}

As you can see the :debug option is not much helpful. Any idea on why or other debugging solutions are welcome

Have a nice day

6 comments

#erlang

0 1543 6

2019-01-30 17:28:56 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

alfredfriedrich

Try using Wireshark, to see whats really going on. Do you see this behaviour (302 vs. 404) with other clients (e.g. curl -iL vs. curl -i)?
Edit:
It seems your :debug is ignored (Invalid option {:verbose, :debug} ignored) see the info message.

Post #1

tangui

Hi, thanks for the response, I didn’t notice that the list was a charlist and indeed one has to use :httpc.set_options/1 to enable debugging.

I didn’t want to try Wireshark because it’s a bit difficult to configure SSL interception, then I noticed that httpc does not check SSL certificates, which is why I eventually switched to HTTPoison.

Post #2

alfredfriedrich

You probably mean :httpc does not check SSL certificates by default (default is {verify, verify_none}).
You have to pass the relevant options. I guess hackney does this by default

Post #3

tangui

Yes, by default

I understand that httpc (and Erlang) doesn’t come with a default trusted CA store. Http libs such as HTTPoison use Certify, which is itself based on Mozilla’s trusted CA list.

Back to httpc: should I enable or disable sending SNIs ({server_name_indication, disable}) since it’s sent cleartext? Should I blacklist these sha1 algs ({signature_algs, [{hash(), ecdsa | rsa | dsa}]})? I have no idea - you need to be an SSL and crypto expert to know - and I guess trying to configure SSL myself will result in a 100% chance of security failure! It’s just surprising there’s no warning in httpc doc because I think most developers will assume that the lib has secure default options.

Post #4

voltone

You’ll want to do something like this:

    :httpc.request(
      :get,
      {'https://badssl.com', []},
      [
        ssl: [
          verify: :verify_peer,
          cacertfile: '/etc/ssl/certs/ca-bundle.crt',
          customize_hostname_check: [
            match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
          ],
          versions: [:'tlsv1.2'],
          depth: 4
        ]
      ],
      []
    )

Change the path to the CA trust store as needed, or call :certifi.cacertfile() if you are relying on Certifi.

Recent OTP versions have reasonable default cipher suite configurations.

Disabling SNI is not recommended: yes, it is sent in the clear, but in many cases it is required by the server to select the appropriate virtual host or upstream server and associated certificate.

Make sure you try your configuration against some of the endpoints listed on badssl.com to verify that bad certificates do indeed cause the connection to fail.

And if you’re going to be at ElixirConf EU in April, come see my talk on this particular topic

Post #5

alfredfriedrich

I can’t think of an easy anwser at the moment besides it's complicated

If you hover over any caption on the erlang manuals, a pencil icon will occur to the left, which lets you fork the repo and propose changes to it. A simple Note box would help I guess. Or some examples in the HTTP Client section of the User’s Guide.

Post #6

— All posts loaded —