Req.get redirects and special characters

Questions / Help
, ,
1

I use the Req library to download a resource. I got the resource from a podcast RSS feed.

enclosure_url = "https://example.com/file 1.mp3"
{:ok, resp} = Req.get(enclosure_url)

I got an error from the Req library. I added an encode Url step, but the resource has some redirects to the final location and in the redirects contains also special characters (eq. a blank).

enclosure_url = "https://example.com/file 1.mp3"
encode_url = URI.encode(enclosure_url)
{:ok, resp} = Req.get(encode_url)

https://example.com/file 1.mp3 → https://cdn.example.com/file 1.mp3

How could I solve this problem?

Remark: I hope the question is not stupid. I’m just starting out with Elixir programming.
I used Elixir with the Req library and without Phoenix.

2

This is a bug in Req, we should be calling your request steps when making the redirect. I’d have to track the history how we ended up there but FWIW here’s a minimal reproduction:

Mix.install([
  {:req, path: "~/src/req"},
  :bandit
])

{:ok, _} =
  Bandit.start_link(
    plug: fn
      conn, _ when conn.request_path == "/" ->
        conn
        |> Plug.Conn.put_resp_header("location", "/foo bar")
        |> Plug.Conn.send_resp(301, "redirecting")

      conn, _ when conn.request_path == "/foo%20bar" ->
        Plug.Conn.send_resp(conn, 200, "ok")
    end,
    port: 8000
  )

Req.new()
|> Req.Request.prepend_request_steps(
  encode_url: fn r ->
    dbg(:encode_url)
    update_in(r.url.path, &(&1 && URI.encode(&1)))
  end
)
|> Req.get!(url: "http://localhost:8000")

and here’s a fix:

diff --git a/lib/req/steps.ex b/lib/req/steps.ex
index 33e332e..8e8fc73 100644
--- a/lib/req/steps.ex
+++ b/lib/req/steps.ex
@@ -1813,6 +1813,8 @@ defmodule Req.Steps do
           |> build_redirect_request(response, location)
           |> Req.Request.put_private(:req_redirect_count, redirect_count + 1)

+        request = put_in(request.current_request_steps, Keyword.keys(request.request_steps))
+
         {request, response_or_exception} = Req.Request.run_request(request)
         Req.Request.halt(request, response_or_exception)
       else

Stay tuned!

4 Likes
3

Thanks for your fast reaction. I will wait for the fix.

4

FWIW one of having steps in Req is you can rearrange and reuse them. So we can work around the bug by building on top of redirect/step like this:

Mix.install([
  :req,
  :bandit
])

{:ok, _} =
  Bandit.start_link(
    plug: fn
      conn, _ when conn.request_path == "/" ->
        conn
        |> Plug.Conn.put_resp_header("location", "/foo bar")
        |> Plug.Conn.send_resp(301, "redirecting")

      conn, _ when conn.request_path == "/foo%20bar" ->
        Plug.Conn.send_resp(conn, 200, "ok")
    end,
    port: 8000
  )

Req.new()
|> then(fn req ->
  update_in(req.response_steps[:redirect], fn fun ->
    fn {req, resp} ->
      req = put_in(req.current_request_steps, Keyword.keys(req.request_steps))
      fun.({req, resp})
    end
  end)
end)
|> Req.Request.prepend_request_steps(
  encode_url: fn r ->
    update_in(r.url.path, &(&1 && URI.encode(&1)))
  end
)
|> Req.get!(url: "http://localhost:8000")
5

I have tried to use the workaround. I encountered a problem if the URL contains query parameters and these contain special characters https://example.com/file 1.mp3 → https://cdn.example.com/file 1.mp3?file=file 1.mp3. The query part is not modified. An attempt is made to access https://cdn.example.com/file%201.mp3?file=file 1.mp3.

But thanks again for the quick help