:httpc redirect issue

erlang

#1

Hi all,

Not an Elixir problem per se but one with Erlang’s httpc module, that is getting me insane. I’m using it in one of my projects trying to avoid adding a dependency for a few http requests.

I’m trying to get a payload at an address, and receive a redirect ({:redirect, false} added for demonstration purpose):

iex(67)> :httpc.request(:get, {'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL?token=[access_token_here]', []}, [{:verbose, :debug}, {:autoredirect, false}], [])

17:28:12.607 [info]  [73, 110, 118, 97, 108, 105, 100, 32, 111, 112, 116, 105, 111, 110, 32, [123, ['verbose', 44, 'debug'], 125], 32, 105, 103, 110, 111, 114, 101, 100, 32, 10]
{:ok,
 {{'HTTP/1.1', 301, 'MOVED PERMANENTLY'},
  [
    {'connection', 'keep-alive'},
    {'date', 'Fri, 11 Jan 2019 14:28:13 GMT'},
    {'location',
     'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=[access_token_here]'},
    {'server', 'cloudflare'},
    {'content-length', '441'},
    {'content-type', 'text/html; charset=utf-8'},
    {'set-cookie',
     '__cfduid=d88330cf7c6165a3dad6797c3cf5e8c171547216892; expires=Sat, 11-Jan-20 14:28:12 GMT; path=/; domain=.fidoalliance.org; HttpOnly'},
    {'x-frame-options', 'SAMEORIGIN'},
    {'x-xss-protection', '1; mode=block'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'cf-ray', '4978118b484e8ed9-DME'}
  ],
  '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>Redirecting...</title>\n<h1>Redirecting...</h1>\n<p>You should be redirected automatically to target URL: <a href="https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=...">https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=d...</a>.  If not click the link.'}}

Let’s try to make a new request from the value of the Location response header (the only difference is the '/' added before the '?'):

iex(68)> :httpc.request(:get, {'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL/?token=[access_token_here]', []}, [{:verbose, :debug}, {:autoredirect, false}], [])

17:30:19.542 [info]  [73, 110, 118, 97, 108, 105, 100, 32, 111, 112, 116, 105, 111, 110, 32, [123, ['verbose', 44, 'debug'], 125], 32, 105, 103, 110, 111, 114, 101, 100, 32, 10]
{:ok,
 {{'HTTP/1.1', 200, 'OK'},
  [
    {'cache-control', 'public, max-age=43200'},
    {'connection', 'keep-alive'},
    {'date', 'Fri, 11 Jan 2019 14:30:20 GMT'},
    {'accept-ranges', 'bytes'},
    {'etag', '"1547064685.0-7052-1005851543"'},
    {'server', 'cloudflare'},
    {'content-length', '7052'},
    {'content-type', 'application/octet-stream'},
    {'expires', 'Sat, 12 Jan 2019 02:30:20 GMT'},
    {'last-modified', 'Wed, 09 Jan 2019 20:11:25 GMT'},
    {'set-cookie',
     '__cfduid=d7e19909933a681371cf8aa0a1c490d191547217019; expires=Sat, 11-Jan-20 14:30:19 GMT; path=/; domain=.fidoalliance.org; HttpOnly'},
    {'x-frame-options', 'SAMEORIGIN'},
    {'content-disposition', 'attachment; filename=t3fyXjAh2Eu7EDsN4tQ6HL'},
    {'x-xss-protection', '1; mode=block'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'cf-ray', '497814a48a104e54-DME'}
  ],
  'eyJhc3NlcnRpb25TY2hlbWUiOiAiVTJGVjFCSU4iLCAiYXR0YW[...]' ++ ...}}

Works fine. But now, when I reenable the auto-redirect (by not setting the :autoredirect option - default is true), I get a 404 error:

iex(69)> :httpc.request(:get, {'https://mds2.fidoalliance.org/metadata/t3fyXjAh2Eu7EDsN4tQ6HL?token=...', []}, [{:verbose, :debug}], [])                         
17:33:29.843 [info]  [73, 110, 118, 97, 108, 105, 100, 32, 111, 112, 116, 105, 111, 110, 32, [123, ['verbose', 44, 'debug'], 125], 32, 105, 103, 110, 111, 114, 101, 100, 32, 10]
{:ok,
 {{'HTTP/1.1', 404, 'NOT FOUND'},
  [
    {'connection', 'keep-alive'},
    {'date', 'Fri, 11 Jan 2019 14:33:31 GMT'},
    {'server', 'cloudflare'},
    {'content-length', '233'},
    {'content-type', 'text/html'},
    {'set-cookie',
     '__cfduid=db7e100503b6a9c7a64fcdb6f851aca681547217210; expires=Sat, 11-Jan-20 14:33:30 GMT; path=/; domain=.fidoalliance.org; HttpOnly'},
    {'x-frame-options', 'SAMEORIGIN'},
    {'x-xss-protection', '1; mode=block'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'cf-ray', '4978194eebbf8eeb-DME'}
  ],
  '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n'}}

As you can see the :debug option is not much helpful. Any idea on why or other debugging solutions are welcome :slight_smile:

Have a nice day


#2

Try using Wireshark, to see whats really going on. Do you see this behaviour (302 vs. 404) with other clients (e.g. curl -iL vs. curl -i)?
Edit:
It seems your :debug is ignored (Invalid option {:verbose, :debug} ignored) see the info message.