Req.get redirects and special characters

dnkbln

I use the Req library to download a resource. I got the resource from a podcast RSS feed.

enclosure_url = "https://example.com/file 1.mp3"
{:ok, resp} = Req.get(enclosure_url)

I got an error from the Req library. I added an encode Url step, but the resource has some redirects to the final location and in the redirects contains also special characters (eq. a blank).

enclosure_url = "https://example.com/file 1.mp3"
encode_url = URI.encode(enclosure_url)
{:ok, resp} = Req.get(encode_url)

https://example.com/file 1.mp3 → https://cdn.example.com/file 1.mp3

How could I solve this problem?

Remark: I hope the question is not stupid. I’m just starting out with Elixir programming.
I used Elixir with the Req library and without Phoenix.

4 comments

#troubleshooting #learning-elixir #req

0 550 4

2024-11-11 17:12:24 UTC

Showing Posts 1 to 4

Show Best Posts
Show All (oldest first)
Show All (newest first)

wojtekmach

Hex Core Team

This is a bug in Req, we should be calling your request steps when making the redirect. I’d have to track the history how we ended up there but FWIW here’s a minimal reproduction:

Mix.install([
  {:req, path: "~/src/req"},
  :bandit
])

{:ok, _} =
  Bandit.start_link(
    plug: fn
      conn, _ when conn.request_path == "/" ->
        conn
        |> Plug.Conn.put_resp_header("location", "/foo bar")
        |> Plug.Conn.send_resp(301, "redirecting")

      conn, _ when conn.request_path == "/foo%20bar" ->
        Plug.Conn.send_resp(conn, 200, "ok")
    end,
    port: 8000
  )

Req.new()
|> Req.Request.prepend_request_steps(
  encode_url: fn r ->
    dbg(:encode_url)
    update_in(r.url.path, &(&1 && URI.encode(&1)))
  end
)
|> Req.get!(url: "http://localhost:8000")

and here’s a fix:

diff --git a/lib/req/steps.ex b/lib/req/steps.ex
index 33e332e..8e8fc73 100644
--- a/lib/req/steps.ex
+++ b/lib/req/steps.ex
@@ -1813,6 +1813,8 @@ defmodule Req.Steps do
           |> build_redirect_request(response, location)
           |> Req.Request.put_private(:req_redirect_count, redirect_count + 1)

+        request = put_in(request.current_request_steps, Keyword.keys(request.request_steps))
+
         {request, response_or_exception} = Req.Request.run_request(request)
         Req.Request.halt(request, response_or_exception)
       else

Stay tuned!

Post #1

dnkbln

Thanks for your fast reaction. I will wait for the fix.

Post #2

wojtekmach

Hex Core Team

FWIW one of having steps in Req is you can rearrange and reuse them. So we can work around the bug by building on top of redirect/step like this:

Mix.install([
  :req,
  :bandit
])

{:ok, _} =
  Bandit.start_link(
    plug: fn
      conn, _ when conn.request_path == "/" ->
        conn
        |> Plug.Conn.put_resp_header("location", "/foo bar")
        |> Plug.Conn.send_resp(301, "redirecting")

      conn, _ when conn.request_path == "/foo%20bar" ->
        Plug.Conn.send_resp(conn, 200, "ok")
    end,
    port: 8000
  )

Req.new()
|> then(fn req ->
  update_in(req.response_steps[:redirect], fn fun ->
    fn {req, resp} ->
      req = put_in(req.current_request_steps, Keyword.keys(req.request_steps))
      fun.({req, resp})
    end
  end)
end)
|> Req.Request.prepend_request_steps(
  encode_url: fn r ->
    update_in(r.url.path, &(&1 && URI.encode(&1)))
  end
)
|> Req.get!(url: "http://localhost:8000")

Post #3

dnkbln

I have tried to use the workaround. I encountered a problem if the URL contains query parameters and these contain special characters https://example.com/file 1.mp3 → https://cdn.example.com/file 1.mp3?file=file 1.mp3. The query part is not modified. An attempt is made to access https://cdn.example.com/file%201.mp3?file=file 1.mp3.

But thanks again for the quick help

Post #4

— All posts loaded —