How to get Partial Content(206) by, passing range header, using Req?

Hello Guys,

I’m building tools using Elixir, Phoenix & LiveView to learn and perhaps have a portfolio.

There’s a LinkedIn post inspector, that shows how a blog post will show up on LinkedIn.

It also shows redirects trail:

image1166×259 25 KB

I recreated the same using Req library, by making a custom step.

FetchExtraMetadata

defmodule DerpyTools.FetchExtraMetadata do
  @moduledoc """
  ## Request Options
    * `:fetch_redirects` - if `true`, gets the metadata related to redirects, like final url, redirects, redirection_trail. Defaults to `false`.
  """
  def attach(%Req.Request{} = request, options \\ []) do
    request
    |> Req.Request.register_options([:fetch_redirects, :print_headers])
    |> Req.Request.merge_options(options)
    |> Req.Request.prepend_response_steps(fetch_redirects: &fetch_redirects/1)
  end

  defp fetch_redirects({%{options: %{fetch_redirects: true}} = request, response}) do
    private =
      request.private
      |> Map.update(
        :trail,
        [{response.status, URI.to_string(request.url)}],
        fn val ->
          [{response.status, URI.to_string(request.url)} | val]
        end
      )

    request = %{request | private: private}

    private =
      response.private
      |> Map.put_new(:redirects, %{
        uri: request.url,
        url: URI.to_string(request.url |> Map.replace(:query, nil)),
        count: Map.get(request.private, :req_redirect_count, 0),
        trail: Map.get(request.private, :trail, [])
      })

    response = %{response | private: private}

    {request, response}
  end

  defp fetch_redirects({%{options: _} = request, response}), do: {request, response}
end

Here’s how I use it:

res =
      Req.new()
      |> FetchExtraMetadata.attach(fetch_redirects: true)
      |> Req.get!(url: url)

IO.inspect(res.private.redirects, label: "Redirect Trail")

image1083×214 33.4 KB

Notice, how I receive 200 instead of 206 shown on LinkedIn’s post inspector.

The HTTP Status Code 206 means that the server is delivering only part of the resource requested by the client due to a range header sent by the client.

I tried sending range header via Req, but I still receive 200 on some links, and for some I get errors, but no 206!

What is the valid value for range header, and how to send it using Req?

I tried: Req.new(range: "bytes=100-200"), but it errors out on certain routes.

[error] GenServer #PID<0.2572.0> terminating
** (stop) :data_error
    :zlib.inflate_nif(#Reference<0.3960474752.3632136194.182349>, 8192, 16384, 0)
    :zlib.dequeue_all_chunks_1/3
    :zlib.inflate/3
    :zlib.gunzip/1
    (elixir 1.15.4) lib/enum.ex:2510: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.15.4) lib/map.ex:916: Map.update!/3
    (req 0.3.11) lib/req/steps.ex:860: Req.Steps.decompress_body/1
    (req 0.3.11) lib/req/request.ex:755: anonymous fn/2 in Req.Request.run_response/2
    (elixir 1.15.4) lib/enum.ex:4830: Enumerable.List.reduce/3
    (elixir 1.15.4) lib/enum.ex:2564: Enum.reduce_while/3
    (req 0.3.11) lib/req/request.ex:683: Req.Request.run/1
    (req 0.3.11) lib/req/steps.ex:1160: Req.Steps.follow_redirects/1
    (req 0.3.11) lib/req/request.ex:755: anonymous fn/2 in Req.Request.run_response/2
    (elixir 1.15.4) lib/enum.ex:4830: Enumerable.List.reduce/3
    (elixir 1.15.4) lib/enum.ex:2564: Enum.reduce_while/3
    (req 0.3.11) lib/req/request.ex:683: Req.Request.run/1
    (req 0.3.11) lib/req/steps.ex:1160: Req.Steps.follow_redirects/1
    (req 0.3.11) lib/req/request.ex:755: anonymous fn/2 in Req.Request.run_response/2
    (elixir 1.15.4) lib/enum.ex:4830: Enumerable.List.reduce/3
    (elixir 1.15.4) lib/enum.ex:2564: Enum.reduce_while/3
Last message: {:analyze_metadata, "http://derpytools.com/hostsfile-mkcert-caddy-achieving-development-production-parity-has-never-been-easier"}

P.S. My project is open source, See: Derpy Tools.
Files to look for:
metadata_analyzer_live.ex
fetch_extra_metadata.ex