drakvuf
I’d like to scrape a website but for some reason, HTTPoison gives me an empty body.
iex> HTTPoison.get(“Origo”)
{:ok,
%HTTPoison.Response{
body: “”,
headers: [
{“Content-length”, “0”},
{“Location”, “Origo”}
],
request: %HTTPoison.Request{
body: “”,
headers: ,
method: :get,
options: ,
params: %{},
url: “Origo”
},
request_url: “Origo”,
status_code: 301
}}
I tried it with HTTPotion too, the same result:
%HTTPotion.Response{
body: “”,
headers: %HTTPotion.Headers{
hdrs: %{
“content-length” => “0”,
“location” => “Origo”
}
},
status_code: 301
}
I checked the site, it is definitely not empty. I tried to fetch it with Nodejs and it works with that.
As far as I understand both of these clients use Hackney and there is an issue about this on the HTTPoisonGithub. But this issue is closed, so I guess this problem shouldn’t exist anymore?
Does anyone have any idea what should I do here?
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security










Showing Posts 3 to 1- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
hubertlepicki
That happens a lot, sometimes you just end up writing question or explain problem to someone and solution comes in, sometimes you actually need 2nd pair of eyes to spot obvious.
drakvuf
OMG. Feel so stupid right now. Thank you so much! I don’t know how I missed the status code…
hubertlepicki
So you are getting a 301 redirect, precisely you are being redirected from http:// to https://.
You can use just http:// in the original URL, or configure HTTPoison to follow redirects like this: httpoison/test/httpoison_test.exs at 3481d567adcb06d129dfec3c69aa87d1719f7a8b · edgurgel/httpoison · GitHub