Req v0.4 is out!
Req v0.4.0 changes headers to be maps, adds request & response streaming, and improves steps.
Change Headers to be Maps
Previously headers were lists of name/value tuples, e.g.:
[{"content-type", "text/html"}]
This is a standard across the ecosystem (with minor difference that some Erlang libraries use charlists instead of binaries.)
There are some problems with this particular choice though:
- We cannot use
headers[name] - We cannot use pattern matching
In short, this representation isn’t very ergonomic to use.
Now headers are maps of string names and lists of values, e.g.:
%{"content-type" => ["text/html"]}
This allows headers[name] usage:
response.headers["content-type"]
#=> ["text/html"]
and pattern matching:
case Req.request!(req) do
%{headers: %{"content-type" => ["application/json" <> _]}} ->
# handle JSON response
end
This is a major breaking change. If you cannot easily update your app or your dependencies, do:
# config/config.exs
config :req, legacy_headers_as_lists: true
This legacy fallback will be removed on Req 1.0.
There are two other changes to headers in this release.
Header names are now case-insensitive in functions like Req.Response.get_header/2.
Trailer headers, or more precisely trailer fields or simply trailers, are now stored in a separate trailers field on the %Req.Response{} struct as long as you use Finch 0.17+.
Add Request Body Streaming
Req v0.4 adds official support for request body streaming by setting the request body to an enumerable. Here’s an example:
iex> stream = Stream.duplicate("foo", 3)
iex> Req.post!("https://httpbin.org/post", body: stream).body["data"]
"foofoofoo"
The enumerable is passed through request steps and they may change it. For example, the compress_body step gzips the request body on the fly.
Add Response Body Streaming
Req v0.4 also adds response body streaming, via the :into option.
Here’s an example where we download the first 20kb (by making a range request, via the put_range step) of Elixir release zip. We stream the response body into a function and can handle each body chunk. The function receives a {:data, data}, {req, resp} and returns a {:cont | :halt, {req, resp}} tuple.
resp =
Req.get!(
url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
range: 0..20_000,
into: fn {:data, data}, {req, resp} ->
IO.inspect(byte_size(data), label: :chunk)
{:cont, {req, resp}}
end
)
# output: 17:07:38.131 [debug] redirecting to https://objects.githubusercontent.com/github-production-release-asset-2e6(...)
# output: chunk: 16384
# output: chunk: 3617
resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> ""
Notice we only stream response body, that is, Req automatically handles HTTP response status and headers. Once the stream is done, Req passes the response through response steps which allows following redirects, retrying on errors, etc. Response body is set to empty string "" which is then ignored by decompress_body, decode_body, and similar steps. If you need to decompress or decode incoming chunks, you need to do that in your custom into: fun function.
As the name :into implies, we can also stream response body into any Collectable. Here’s a similar snippet to above where we stream to a file:
resp =
Req.get!(
url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
range: 0..20_000,
into: File.stream!("elixit-otp-26.zip.1")
)
# output: 17:07:38.131 [debug] redirecting to (...)
resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> %File.Stream{}
Full CHANGELOG
-
Change
request.headersandresponse.headersto be maps. -
Ensure
request.headersandresponse.headersare downcased.Per RFC 9110: HTTP Semantics, HTTP headers should be case-insensitive. However, per RFC 9113: HTTP/2 headers must be sent downcased.
Req headers are now stored internally downcased and all accessor functions like
Req.Response.get_header/2are downcasing the given header name. -
Add
trailersfield toReq.Responsestruct. Trailer field is only filled in on Finch 0.17+. -
Make
request.registered_optionsinternal representation private. -
Make
request.optionsinternal representation private.Currently
request.optionsfield is a map but it may change in the future. One possible future change is using keywords lists internally which would allow, for example,Req.new(params: [a: 1]) |> Req.update(params: [b: 2])to keep duplicate:paramsinrequest.optionswhich would then allow to decide the duplicate key semantics on a per-step basis. And so, for example,put_paramswould merge params but most steps would simply use the first value.To have some room for manoeuvre in the future we should stop pattern matching on
request.options. Callingrequest.options[key],put_in(request.options[key], value), andupdate_in(request.options[key], fun)is allowed. -
Fix typespecs for some functions
-
Deprecate
outputstep in favour ofinto: File.stream!(path). -
Rename
follow_redirectsstep toredirect -
redirect: Rename:follow_redirectsoption to:redirect. -
redirect: Rename:location_trustedoption to:redirect_trusted. -
redirect: Change HTTP request method to GET only on POST requests that result in 301..303.Previously we were changing the method to GET for all 3xx except 307 and 308.
-
decompress_body: Remove support fordeflatecompression (which was broken) -
decompress_body: Don’t crash on unknown codec -
decompress_body: Fix handling HEAD requests -
decompress_body: Re-calculatecontent-lengthheader after decompresion -
decompress_body: Removecontent-encodingheader after decompression -
decode_body: Do not decode response withcontent-encodingheader -
run_finch: Add:inet6option -
retry: Supportretry: :safe_transientwhich retries HTTP 408/429/500/502/503/504 or exceptions withreasonfield set to:timeout/:econnrefused.:safe_transientis the new default retry mode. (Previously we retried on 408/429/5xx and any exception.) -
retry: Supportretry: :transientwhich is the same as:safe_transientexcept it retries on all HTTP methods -
retry: Useretry-afterheader value on HTTP 503 Service Unavailable. Previously only HTTP 429 Too Many Requests was using this header value. -
retry: Supportretry: &fun/2. The function receivesrequest, response_or_exceptionand returns either:-
true- retry with the default delay -
{:delay, milliseconds}- retry with the given delay -
false/nil- don’t retry
-
-
retry: Deprecateretry: :safein favour ofretry: :safe_transient -
retry: Deprecateretry: :neverin favour ofretry: false -
Req.request/2: Improve error message on invalid arguments -
Req.update/2: Do not duplicate headers -
Req.update/2: Merge:params -
Req.Request: Fix displaying redacted basic authentication
