drakvuf
I’d like to scrape a website but for some reason, HTTPoison gives me an empty body.
iex> HTTPoison.get(“Origo”)
{:ok,
%HTTPoison.Response{
body: “”,
headers: [
{“Content-length”, “0”},
{“Location”, “Origo”}
],
request: %HTTPoison.Request{
body: “”,
headers: ,
method: :get,
options: ,
params: %{},
url: “Origo”
},
request_url: “Origo”,
status_code: 301
}}
I tried it with HTTPotion too, the same result:
%HTTPotion.Response{
body: “”,
headers: %HTTPotion.Headers{
hdrs: %{
“content-length” => “0”,
“location” => “Origo”
}
},
status_code: 301
}
I checked the site, it is definitely not empty. I tried to fetch it with Nodejs and it works with that.
As far as I understand both of these clients use Hackney and there is an issue about this on the HTTPoisonGithub. But this issue is closed, so I guess this problem shouldn’t exist anymore?
Does anyone have any idea what should I do here?
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security










Showing Posts 1 to 3- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
hubertlepicki
So you are getting a 301 redirect, precisely you are being redirected from http:// to https://.
You can use just http:// in the original URL, or configure HTTPoison to follow redirects like this: httpoison/test/httpoison_test.exs at 3481d567adcb06d129dfec3c69aa87d1719f7a8b · edgurgel/httpoison · GitHub
drakvuf
OMG. Feel so stupid right now. Thank you so much! I don’t know how I missed the status code…
hubertlepicki
That happens a lot, sometimes you just end up writing question or explain problem to someone and solution comes in, sometimes you actually need 2nd pair of eyes to spot obvious.