Help me convert this curl request to a Req.get! request?

Questions / Help
1

This curl request works:

curl --location --request GET 'https://www.target.com/s?searchTerm=sonic+frontiers&sortBy=relevance&category=5xtg5&facetedValue=nqueqc6d44fZ54jftZdq4mnZ2q830p6d44f&moveTo=product-list-grid' \
--header 'sec-ch-ua: "Brave";v="107", "Chromium";v="107", "Not=A?Brand";v="24"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "Windows"' \
--header 'Upgrade-Insecure-Requests: 1' \
--header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' \
--header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8' \
--header 'Sec-GPC: 1' \
--header 'Cookie: TealeafAkaSid=JA-JSAXRCLjKYhjV9IXTzYUbcV1Lnhqf; sapphire=1; visitorId=0184E4601D5A020183FFBB13380347CE; GuestLocation=33196|25.660|-80.440|FL|US'

I attempted to convert this to a Req.get! request, but I get an accessDenied error from the server.

    req_url =
      "https://www.target.com/s?searchTerm=sonic+frontiers&sortBy=relevance&category=5xtg5&facetedValue=nqueqc6d44fZ54jftZdq4mnZ2q830p6d44f&moveTo=product-list-grid"

    headers = %{
      "sec-ch-ua" => "\"Brave\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
      "sec-ch-ua-mobile" => "?0",
      "sec-ch-ua-platform" => "Windows",
      "Upgrade-Insecure-Requests" => 1,
      "User-Agent" =>
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
      "Accept" =>
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
      "Sec-GPC" => 1,
      "Cookie" =>
        "TealeafAkaSid=JA-JSAXRCLjKYhjV9IXTzYUbcV1Lnhqf; sapphire=1; visitorId=0184E4601D5A020183FFBB13380347CE; GuestLocation=33196|25.744|-80.292|FL|US"
    }

    html = Req.get!(req_url, headers: headers, follow_redirects: true)

Error:

%Req.Response{
  status: 403,
  headers: [
    {"connection", "close"},
    {"content-length", "32"},
    {"server", "Varnish"},
    {"retry-after", "0"},
    {"content-type", "application/json"},
    {"accept-ranges", "bytes"},
    {"date", "Tue, 06 Dec 2022 00:28:18 GMT"},
    {"set-cookie",
     "TealeafAkaSid=JA-JSAXRCLjKYhjV9IXTzYUbcV1Lnhqf; Expires=Thu, 05 Jan 2023 00:28:18 GMT; Path=/; Domain=target.com;"},
    {"clientgeo", "US"},
    {"clientip", "75.33.221.151"},
    {"x-frame-options", "SAMEORIGIN"},
    {"content-security-policy", "frame-ancestors 'self' https://*.target.com;"},
    {"x-xss-protection", "1; mode=block"},
    {"x-content-type-options", "nosniff"},
    {"referrer-policy", "no-referrer-when-downgrade"},
    {"strict-transport-security", "max-age=31536000; includeSubDomains"},
    {"cache-control", "no-cache"}
  ],
  body: %{"accessDenied" => "75.33.221.151"},
  private: %{}
}

What am I doing wrong trying to convert this working curl into an Elixir Req.get! request? Thank you!

3 Likes
2

A workaround for now is I can just call curl with System.cmd but I don’t know how that would affect the performance of my app.

Does anyone know the downsides of doing it this way?

command = "curl"

params = [
  "--location",
  "--request",
  "GET",
  "https://www.target.com/s?searchTerm=#{search_term}&sortBy=relevance&category=5xtg5&facetedValue=nqueqc6d44fZ54jftZdq4mnZ2q830p6d44f&moveTo=product-list-grid",
  "--header",
  "sec-ch-ua: \"Brave\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
  "--header",
  "sec-ch-ua-mobile: ?0",
  "--header",
  "sec-ch-ua-platform: \"Windows\"",
  "--header",
  "Upgrade-Insecure-Requests: 1",
  "--header",
  "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
  "--header",
  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
  "--header",
  "Sec-GPC: 1",
  "--header",
  "Cookie: TealeafAkaSid=JA-JSAXRCLjKYhjV9IXTzYUbcV1Lnhqf; sapphire=1; visitorId=0184E4601D5A020183FFBB13380347CE; GuestLocation=33196|25.660|-80.292|FL|US"
]
3

Hi,

in your non working example you are creating the headers as a map. But according to the Req documentation your need to define the headers as a list of tuples of two binaries.

headers = [{binary, binary}]

Hope that helped?!

4

I get accessDenied for the curl you’ve provided.

Maybe the problem is that Req doesn’t override useragent header. It just duplicates it.

For example,

Req.get "http://localhost:3000", headers: %{"User-Agent" => "MOZ"}

Returns

GET / HTTP/1.1
host: localhost:3000
user-agent: MOZ
user-agent: req/0.3.2
accept-encoding: gzip, deflate

Just tested. Map works correctly

5

You can just use user_agent as an option, not as a header

For example,

Req.get!("http://localhost:3000", user_agent: "foo")

GET / HTTP/1.1
host: localhost:3000
user-agent: foo
accept-encoding: gzip, deflate
1 Like
6

Req treats headers case sensitively and expects (like http2 requires) folks to use lower-case. It sets user-agent by default, so if you set User-Agent too, you’d get both.

That’s not the root cause though. It looks like Mint is sending a request that the server does not allow. The headers don’t seem to matter.

Mix.install([:hackney, :mint, :castore])

url =
  "https://www.target.com/s?searchTerm=sonic+frontiers&sortBy=relevance&category=5xtg5&facetedValue=nqueqc6d44fZ54jftZdq4mnZ2q830p6d44f&moveTo=product-list-grid"

{:ok, status, _headers, _ref} = :hackney.request(url)
dbg(status)
# status #=> 200

{:ok, conn} = Mint.HTTP.connect(:https, "www.target.com", 443)
url = String.trim_leading(url, "https://www.target.com")
{:ok, conn, _ref} = Mint.HTTP.request(conn, "GET", url, [], "")

receive do
  message ->
    {:ok, _conn, responses} = Mint.HTTP.stream(conn, message)
    responses
end
|> dbg

# {:status, #Reference<0.944912404.956301313.218177>, 403}
# (...)

if someone could investigate this further and maybe report to GitHub - elixir-mint/mint: Functional HTTP client for Elixir with support for HTTP/1 and HTTP/2 🌱 that’d be appreciated.

3 Likes
7

You can use 🥌 CurlReq — CurlReq v0.100.0 to convert a cURL command to a Req.Request

Mix.install([
  {:req, "~> 0.5.8"},
  {:curl_req, "~> 0.100.0"}
])

"""
curl --location --request GET 'https://www.target.com/s?searchTerm=sonic+frontiers&sortBy=relevance&category=5xtg5&facetedValue=nqueqc6d44fZ54jftZdq4mnZ2q830p6d44f&moveTo=product-list-grid' \
--header 'sec-ch-ua: "Brave";v="107", "Chromium";v="107", "Not=A?Brand";v="24"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "Windows"' \
--header 'Upgrade-Insecure-Requests: 1' \
--header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' \
--header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8' \
--header 'Sec-GPC: 1' \
--header 'Cookie: TealeafAkaSid=JA-JSAXRCLjKYhjV9IXTzYUbcV1Lnhqf; sapphire=1; visitorId=0184E4601D5A020183FFBB13380347CE; GuestLocation=33196|25.660|-80.440|FL|US'
"""
|> CurlReq.from_curl()
#|> Req.request!()

Result:

%Req.Request{
  method: :get,
  url: URI.parse("https://www.target.com/s?searchTerm=sonic+frontiers&sortBy=relevance&category=5xtg5&facetedValue=nqueqc6d44fZ54jftZdq4mnZ2q830p6d44f&moveTo=product-list-grid"),
  headers: %{
    "accept" => ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],
    "cookie" => ["GuestLocation=33196|25.660|-80.440|FL|US; TealeafAkaSid=JA-JSAXRCLjKYhjV9IXTzYUbcV1Lnhqf; sapphire=1; visitorId=0184E4601D5A020183FFBB13380347CE"],
    "sec-ch-ua" => ["\"Brave\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\""],
    "sec-ch-ua-mobile" => ["?0"],
    "sec-ch-ua-platform" => ["\"Windows\""],
    "sec-gpc" => ["1"],
    "upgrade-insecure-requests" => ["1"],
    "user-agent" => ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"]
  },
  body: nil,
  options: %{redirect: true},
  halted: false,
  adapter: &Req.Steps.run_finch/1,
  request_steps: [],
  response_steps: [redirect: &Req.Steps.redirect/1],
  error_steps: [],
  private: %{}
}
3 Likes