Req: how to debug Req.TransportError{reason: :closed} for proxy?

Questions / Help
1

I’m trying to understand why this code works for one provider but not for another

[
  url: "https://ip.smartproxy.com/json",
  connect_options: [
    proxy: {:http, "dc.smartproxy.com", 10001, []},
    transport_opts: [verify: :verify_none],
    proxy_headers: [
      {"Proxy-Authorization", "Basic " <> Base.encode64("username:password")}
    ]
  ],
  retry: false
]
|> Req.get()
# {:error, %Req.TransportError{reason: :closed}}

curl works for both providers:

curl -v -U "username:password" -x "dc.smartproxy.com:10001" "https://ip.smartproxy.com/json"

Is there a verbose option for Req similar to curl --verbose?

2

I believe the -u option is just prepending the username and password to the url like https://username:password@ip.smartproxy.com/json

Not sure what the Proxy-Authorization header is for, but that’s not present in the curl command. Additionally the -x parameter should be used to declare the http method, which you don’t seem to be doing

3

I think you’ve mixed up your casing for -U and -x - which are frustratingly similar for quite different purposes! The problem of highly configurable software.

I’d double check the headers sent with curl. I’m not exactly sure how to log request headers or why this isn’t working, unfortunately.

4

curl is correct and working

-U, --proxy-user <user:password>
-x, --proxy [protocol://]host[:port]

Elixir code works with Brightdata but does not function with the Smartproxy provider. While both providers work via curl

The question is how to debug such problems? Is there a way to print debug logs like curl -v does?

❯ curl -v -U "username:password" -x "brd.superproxy.io:1234" "https://geo.brdtest.com/welcome.txt?product=dc&method=native"
* Host brd.superproxy.io:1234 was resolved.
* IPv6: (none)
* IPv4: 1.2.3.4, 1.2.3.5
*   Trying 1.2.3.4:1234...
* Connected to brd.superproxy.io (1.2.3.4) port 1234
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Proxy auth using Basic with user 'username'
* Establish HTTP proxy tunnel to geo.brdtest.com:443
> CONNECT geo.brdtest.com:443 HTTP/1.1
> Host: geo.brdtest.com:443
> Proxy-Authorization: Basic <base64-encoded-username-and-password>
> User-Agent: curl/8.7.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< x-luminati-ip: 136.227.174.143
...
5

I fixed it by passing protocols: [:http2] to the connect_options.

Here is the working version:

[
  url: "https://ip.smartproxy.com/json",
  connect_options: [
    proxy: {:http, "dc.smartproxy.com", 1234, []},
    protocols: [:http2],
    transport_opts: [verify: :verify_none],
    proxy_headers: [
      {"Proxy-Authorization", "Basic " <> Base.encode64("username:password")}
    ]
  ],
  retry: false
]
|> Req.get()

But I’m still not sure how to debug similar problems.

1 Like
6

My bad, I did not see those options when I checked my curl version, I just assumed the casing was incorrect.