HTTPoison : Get body of redirected location target

Nicolas

Hi,

I have this URL https://beerfactory.org/api/v1/apps which redirects (HTTP 301) to https://mastodon.beerfactory.org/api/v1/apps.

When using HTTPoison to post of this URL with the following code to follow the redirection:

HTTPoison.request(:post, "https://beerfactory.org/api/v1/apps", "", [], [follow_redirect: true])

I get the following result :

{:ok,
 %HTTPoison.AsyncResponse{
   id: {:maybe_redirect, 301,
    [
      {"Server", "nginx/1.14.0 (Ubuntu)"},
      {"Date", "Mon, 29 Oct 2018 21:45:22 GMT"},
      {"Content-Type", "text/html"},
      {"Content-Length", "194"},
      {"Connection", "keep-alive"},
      {"Location", "https://mastodon.beerfactory.org/api/v1/apps"}
    ],
    {:client, {1540, 849522, 427514}, {:metrics_ng, :metrics_dummy},
     :hackney_ssl, 'beerfactory.org', 443, "beerfactory.org",
     [max_redirect: 5, follow_redirect: true],
     {:sslsocket, {:gen_tcp, #Port<0.7>, :tls_connection, :undefined},
      [#PID<0.204.0>, #PID<0.203.0>]},
     {:default, #Reference<0.4243972227.2875195393.28572>,
      {'beerfactory.org', 443, :hackney_ssl}, #PID<0.199.0>, :hackney_ssl},
     #Reference<0.4243972227.2875195393.28572>, true, :hackney_pool, 5000, true,
     5, false, 5, nil, nil,
     {:hparser, :response, 4096, 10, 0, :on_body,
      "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx/1.14.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n",
      {1, 1}, "", [], 194, "", "keep-alive", "text/html",
      "https://mastodon.beerfactory.org/api/v1/apps", :waiting},
     {6,
      {:dict, 6, 16, 16, 8, 80, 48,
       {[], [], [], [], [], [], [], [], [], [], [], ...},
       {{[["content-length", {3, "Content-Length", "194"}]],
         [
           [
             "location",
             {5, "Location", "https://mastodon.beerfactory.org/api/v1/apps"}
           ]
         ], [["date", {1, "Date", "Mon, 29 Oct 2018 21:45:22 GMT"}]], [], [],
         [["server", {...}]], [], [], [], ...}}}}, :connected, :waiting, nil,
     :normal, false, false, false, :undefined, false, &:hackney_request.send/2,
     :waiting, nil, 4096, "", [], {1, 1}, 194, nil, nil, "POST", "/api/v1/apps",
     ...}}
 }}

Is there a way I can get the redirection target body from this result ?

Thanks.

4 comments

#httpoison

0 4209 4

2018-10-30 08:02:24 UTC

Showing Posts 1 to 4

Show Best Posts
Show All (oldest first)
Show All (newest first)

Nicolas

Note that when trying the same code with another endpoint which is also redirected but with GET method the redirection works, so may be the automatic redirection works differently with the POST method.

details:

HTTPoison.request!(:get, "https://beerfactory.org/", "", [], [follow_redirect: false])                             
%HTTPoison.Response{
  body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx/1.14.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n",
  headers: [
    {"Server", "nginx/1.14.0 (Ubuntu)"},
    {"Date", "Mon, 29 Oct 2018 22:23:32 GMT"},
    {"Content-Type", "text/html"},
    {"Content-Length", "194"},
    {"Connection", "keep-alive"},
    {"Location", "https://mastodon.beerfactory.org/"}
  ],
  request: %HTTPoison.Request{
    body: "",
    headers: [],
    method: :get,
    options: [follow_redirect: false],
    params: %{},
    url: "https://beerfactory.org/"
  },
  request_url: "https://beerfactory.org/",
  status_code: 301
}

whereas:

HTTPoison.request!(:get, "https://beerfactory.org/", "", [], [follow_redirect: true]) 
%HTTPoison.Response{
  body: "<!DOCTYPE html>\n<html lang='en'>\n<head>\n<meta charset='utf-8'>\n<meta content='width=device-width, initial-scale=1' name='viewport'>\n<link href='/favicon.ico' rel='icon' type='image/x-icon'>\n<link href='/apple-touch-icon.png' rel='apple-touch-icon' sizes='180x180'>\n<link color='#2B90D9' href='/mask-icon.svg' rel='mask-icon'>\n<link href='/manifest.json' rel='manifest'>\n<meta content='/browserconfig.xml' name='msapplication-config'>\n<meta content='#282c37' name='theme-color'>\n<meta content='yes' name='apple-mobile-web-app-capable'>\n<title>beerfactory.org - Mastodon Beerfactory</title>\n<link rel=\"stylesheet\" media=\"all\" href=\"/packs/common-66639ea10b9b1e8d7a8f.css\" />\n<link rel=\"stylesheet\" media=\"all\" href=\"/packs/default-46f44143aecc20b96d31.css\" />\n<script src=\"/packs/common-0400b2a12246408ec6f5.js\" crossorigin=\"anonymous\"></script>\n<script src=\"/packs/locale_en-f0ba7181de296154606a.js\" crossorigin=\"anonymous\"></script>\n<meta name=\"csrf-param\" content=\"authenticity_token\" />\n<meta name=\"csrf-token\" content=\"Nvv30rr5x7bwX93SWFvBzFtgyKY6FmVL27jeTwZEJRQURBBENcEuT23EgF7hNB/MFntbG9biydQbS1LHJl8ePA==\" />\n<link href='https://mastodon.beerfactory.org/about' rel='canonical'>\n<script id='initial-state' type='application/json'>{\"meta\":{\"streaming_api_base_url\":\"wss://mastodon.beerfactory.org\",\"access_token\":null,\"locale\":\"en\",\"domain\":\"beerfactory.org\",\"admin\":null,\"search_enabled\":false,\"version\":\"2.5.2\",\"invites_enabled\":false},\"compose\":{},\"accounts\":{},\"media_attachments\":{\"accept_content_types\":[\".jpg\",\".jpeg\",\".png\",\".gif\",\".webm\",\".mp4\",\".m4v\",\".mov\",\"image/jpeg\",\"image/png\",\"image/gif\",\"video/webm\",\"video/mp4\",\"video/quicktime\"]},\"settings\":{\"known_fediverse\":true},\"push_subscription\":null}</script>\n<script src=\"/packs/about-c5f040b1e03e645e51f7.js\" crossorigin=\"anonymous\"></script>\n<meta content=\"Mastodon hosted on beerfactory.org\" property=\"og:site_name\" />\n<meta content=\"https://mastodon.beerfactory.org/about\" property=\"og:url\" />\n<meta content=\"website\" property=\"og:type\" />\n<meta content=\"Mastodon Beerfactory\" property=\"og:title\" />\n<meta content=\"[FR] Cette instance mastodon, ouverte librement à tous, a pour objectif de faciliter les échanges entre amateurs de bières et brasseurs amateurs (mais pas que). Les quelques règles applicables sont décrites ci-dessous.\r\n[EN] This mastodon instance, free and open for all, aims to facilitate exchanges between beer lovers and home brewers (but not only). The few applicables rules are shown below.\" property=\"og:description\" />\n<meta content=\"https://mastodon.beerfactory.org/packs/preview-9a17d32fc48369e8ccd910a75260e67d.jpg\" property=\"og:image\" />\n<meta content=\"1200\" property=\"og:image:width\" />\n<meta content=\"630\" property=\"og:image:height\" />\n<meta content=\"summary_large_image\" property=\"twitter:card\" />\n\n\n</head>\n<body class='with-modals theme-default no-reduce-motion'>\n<div class='landing-page alternative'>\n<div class='container'>\n<div class='grid'>\n<div class='column-0'>\n<div class='brand'>\n<a href=\"https://mastodon.beerfactory.org/\"><img alt=\"Mastodon\" src=\"/packs/logo_full-96e7a97fe469f75a23a74852b2478fa3.svg\" />\n</a></div>\n</div>\n<div class='column-1'>\n<div class='landing-page__forms'>\n<div class='brand'>\n<a href=\"https://mastodon.beerfactory.org/\"><img alt=\"Mastodon\" src=\"/packs/logo_full-96e7a97fe469f75a23a74852b2478fa3.svg\" />\n</a></div>\n<form class=\"simple_form new_user\" id=\"new_user\" novalidate=\"novalidate\" action=\"/auth\" accept-charset=\"UTF-8\" method=\"post\"><input name=\"utf8\" type=\"hidden\" value=\"&#x2713;\" /><input type=\"hidden\" name=\"authenticity_token\" value=\"EnVSCnyuwQc4K4K56lCnWf84JZgoM6wBdl1THOZ0/eA6oV75Hc2ghlfK/d2d6Con6RLCr/EGD+Er7Gy94/Zw0g==\" /><div class='input-with-append'>\n<div class=\"input string required user_account_username\"><input aria-label=\"Username\" autocomplete=\"off\" class=\"string required\" autofocus=\"autofocus\" required=\"required\" aria-required=\"true\" placeholder=\"Username\" type=\"text\" value=\"\" name=\"user[account_attributes][username]\" id=\"user_account_attributes_username\" /></div>\n<div class='append'>\n@beerfactory.org\n</div>\n</div>\n<div class=\"i" <> ...,
  headers: [
    {"Date", "Mon, 29 Oct 2018 22:24:05 GMT"},
    {"Content-Type", "text/html; charset=utf-8"},
    {"Transfer-Encoding", "chunked"},
    {"Connection", "keep-alive"},
    {"Vary", "Accept-Encoding"},
    {"Server", "Mastodon"},
    {"X-Frame-Options", "DENY"},
    {"X-Content-Type-Options", "nosniff"},
    {"X-XSS-Protection", "1; mode=block"},
    {"Vary", "Accept-Encoding"},
    {"ETag", "W/\"07c7deb59b92d1af3d22160a27fbbc85\""},
    {"Cache-Control", "max-age=0, private, must-revalidate"},
    {"Set-Cookie",
     "_mastodon_session=Yp8sHLxiLyNGlC%2FQiMyMerVQ8%2BCs1enX%2BZo2noRyNr73fNkYE2iUfciNUnq3BRQrCuqVDokwWnLWnvTV1HI%2F79tDtafMGA8GdC4QL5j9UctyPFjzYwXKjPutAO22GS0HBXyZqHnQCrEhDaJwlpRTROJ3d8ADjn%2FOW8QIuY816bkIgKEJuNowjw%3D%3D--fUHmKYnWNTl7wVjt--kefIlYFLI9uNAaEkOeDAfA%3D%3D; path=/; secure; HttpOnly"},
    {"X-Request-Id", "df27847d-0519-44f1-98d7-bbcf617b5ab2"},
    {"X-Runtime", "0.069550"},
    {"Strict-Transport-Security", "max-age=31536000"}
  ],
  request: %HTTPoison.Request{
    body: "",
    headers: [],
    method: :get,
    options: [follow_redirect: true],
    params: %{},
    url: "https://beerfactory.org/"
  },
  request_url: "https://beerfactory.org/",
  status_code: 200
}

Post #1

frigidcode

POST redirects should use 307 according to the spec. 301 would require prompting the user for anything other than HEAD/GET verbs. It appears HTTPoison is acting as it should by not following the POST redirect.

As for your first question I will play with it and see if I can get the location parsed out.

Post #2

frigidcode

{:ok, resp} = HTTPoison.post("https://beerfactory.org/api/v1/apps", "", [], [follow_redirect: true])

resp.id |> elem(2) |> Enum.find(fn v -> elem(v,0) == "Location" end)

{"Location", "https://mastodon.beerfactory.org/api/v1/apps"}

quick solution, might be cleaner ones available (very tired ATM)

Post #3

Nicolas

OK, reading hackney documentation and source code it appears that hackney doesn’t follow 301/302 redirection unless you add force_redirect: true in the options:
In my example, the redirection is followed when both follow_redirect and force_redirect are specified:

HTTPoison.request(:post, "https://beerfactory.org/api/v1/apps", "", [], [follow_redirect: true, hackney: [{:force_redirect, true}]])                          
{:ok,
 %HTTPoison.Response{
   body: "{\"error\":\"Validation failed: Application name can't be blank, Redirect URI can't be blank\"}",
   headers: [
     {"Date", "Tue, 30 Oct 2018 08:00:42 GMT"},
     {"Content-Type", "application/json; charset=utf-8"},
     {"Transfer-Encoding", "chunked"},
     {"Connection", "keep-alive"},
     {"Server", "Mastodon"},
     {"X-Frame-Options", "DENY"},
     {"X-Content-Type-Options", "nosniff"},
     {"X-XSS-Protection", "1; mode=block"},
     {"X-RateLimit-Limit", "7500"},
     {"X-RateLimit-Remaining", "7499"},
     {"X-RateLimit-Reset", "2018-10-30T08:05:00.943912Z"},
     {"Vary", "Accept-Encoding, Origin"},
     {"Cache-Control", "no-cache"},
     {"X-Request-Id", "535f3d45-e48c-4a32-8a5a-92ccb72d6f21"},
     {"X-Runtime", "0.028489"}
   ],
   request: %HTTPoison.Request{
     body: "",
     headers: [],
     method: :post,
     options: [follow_redirect: true, hackney: [force_redirect: true]],
     params: %{},
     url: "https://beerfactory.org/api/v1/apps"
   },
   request_url: "https://beerfactory.org/api/v1/apps",
   status_code: 422
 }}

422 status is due to the bad request format used for this example.

Post #4

— All posts loaded —