HTTPS request using Req and DNS over HTTPS

Questions / Help
, ,
1

Hey everyone,

I’m still pretty new to Elixir so I don’t have any experience with any HTTP library, but ran into an issue using Req. Since I was able to solve my problem using HTTPoisen, this post is more out of curiosity, whether there is a solution using Req and what I might have missed/misunderstood.

Problem

I want to fetch a JSON file from a HTTPS URL using a DNS over HTTPS service (Cloudflare’s, in this case). The JSON file might be hosted at the resolved IP address, or it might redirect yet another time.

I put up some sample projects for my own testing (they are still live):
https://vercel-redirect-test-json.vercel.app/some.json
redirects to
https://vercel-static-test-json.vercel.app/some.json
where some.json is hosted.

Using cURL and HTTPoison works!

In the terminal I use dig to look up the IP address and can resolve the JSON file using cURL with the --resolve option:

curl -L --resolve vercel-redirect-test-json.vercel.app:443:76.76.21.61 \
    https://vercel-redirect-test-json.vercel.app/some.json

I was able to get the request working using HTTPoison and hackney:

resolved_ip = "76.76.21.61" # this would be the result of the dns-query to Cloudflare
domain = "vercel-redirect-test-json.vercel.app"
resolve_option = {:resolve, [{domain, 443, ip}]

HTTPoison.get(
  "https://#{domain}/some.json",
  [],
  follow_redirect: true,
  hackney: [options: [resolve_option]]
)

How to use Req?

I did not find anything like a resolve option, and while I can resolve the “statically” hosted file by specifically providing the hostname to avoid any SSL errors,

resolved_ip = "76.76.21.61" # this would be the result of the dns-query to Cloudflare
domain = "vercel-static-test-json.vercel.app"
request =
  Req.new(
    method: :get,
    url: "https://#{ip}/some.json",
    connect_options: [hostname: domain]
  )

Req.get(request) # this works, the response has the JSON object!

I get repeated debug logs when trying to use the redirected version https://vercel-redirect-test-json.vercel.app/some.json

[debug] redirecting to https://vercel-static-test-json.vercel.app/some.json

and error out in the end with ** (RuntimeError) too many redirects (10).

What am I doing wrong?