Hello Guys,
I’m building tools using Elixir, Phoenix & LiveView to learn and perhaps have a portfolio.
There’s a LinkedIn post inspector, that shows how a blog post will show up on LinkedIn.
It also shows redirects trail:
I recreated the same using Req library, by making a custom step.
FetchExtraMetadata
defmodule DerpyTools.FetchExtraMetadata do
@moduledoc """
## Request Options
* `:fetch_redirects` - if `true`, gets the metadata related to redirects, like final url, redirects, redirection_trail. Defaults to `false`.
"""
def attach(%Req.Request{} = request, options \\ []) do
request
|> Req.Request.register_options([:fetch_redirects, :print_headers])
|> Req.Request.merge_options(options)
|> Req.Request.prepend_response_steps(fetch_redirects: &fetch_redirects/1)
end
defp fetch_redirects({%{options: %{fetch_redirects: true}} = request, response}) do
private =
request.private
|> Map.update(
:trail,
[{response.status, URI.to_string(request.url)}],
fn val ->
[{response.status, URI.to_string(request.url)} | val]
end
)
request = %{request | private: private}
private =
response.private
|> Map.put_new(:redirects, %{
uri: request.url,
url: URI.to_string(request.url |> Map.replace(:query, nil)),
count: Map.get(request.private, :req_redirect_count, 0),
trail: Map.get(request.private, :trail, [])
})
response = %{response | private: private}
{request, response}
end
defp fetch_redirects({%{options: _} = request, response}), do: {request, response}
end
Here’s how I use it:
res =
Req.new()
|> FetchExtraMetadata.attach(fetch_redirects: true)
|> Req.get!(url: url)
IO.inspect(res.private.redirects, label: "Redirect Trail")
Notice, how I receive 200 instead of 206 shown on LinkedIn’s post inspector.
The HTTP Status Code 206 means that the server is delivering only part of the resource requested by the client due to a range header sent by the client.
I tried sending range header via Req, but I still receive 200 on some links, and for some I get errors, but no 206!
What is the valid value for range header, and how to send it using Req?
I tried: Req.new(range: "bytes=100-200"), but it errors out on certain routes.
[error] GenServer #PID<0.2572.0> terminating
** (stop) :data_error
:zlib.inflate_nif(#Reference<0.3960474752.3632136194.182349>, 8192, 16384, 0)
:zlib.dequeue_all_chunks_1/3
:zlib.inflate/3
:zlib.gunzip/1
(elixir 1.15.4) lib/enum.ex:2510: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir 1.15.4) lib/map.ex:916: Map.update!/3
(req 0.3.11) lib/req/steps.ex:860: Req.Steps.decompress_body/1
(req 0.3.11) lib/req/request.ex:755: anonymous fn/2 in Req.Request.run_response/2
(elixir 1.15.4) lib/enum.ex:4830: Enumerable.List.reduce/3
(elixir 1.15.4) lib/enum.ex:2564: Enum.reduce_while/3
(req 0.3.11) lib/req/request.ex:683: Req.Request.run/1
(req 0.3.11) lib/req/steps.ex:1160: Req.Steps.follow_redirects/1
(req 0.3.11) lib/req/request.ex:755: anonymous fn/2 in Req.Request.run_response/2
(elixir 1.15.4) lib/enum.ex:4830: Enumerable.List.reduce/3
(elixir 1.15.4) lib/enum.ex:2564: Enum.reduce_while/3
(req 0.3.11) lib/req/request.ex:683: Req.Request.run/1
(req 0.3.11) lib/req/steps.ex:1160: Req.Steps.follow_redirects/1
(req 0.3.11) lib/req/request.ex:755: anonymous fn/2 in Req.Request.run_response/2
(elixir 1.15.4) lib/enum.ex:4830: Enumerable.List.reduce/3
(elixir 1.15.4) lib/enum.ex:2564: Enum.reduce_while/3
Last message: {:analyze_metadata, "http://derpytools.com/hostsfile-mkcert-caddy-achieving-development-production-parity-has-never-been-easier"}
P.S. My project is open source, See: Derpy Tools.
Files to look for:
metadata_analyzer_live.ex
fetch_extra_metadata.ex
