Req v0.4 is out!

Req v0.4.0 changes headers to be maps, adds request & response streaming, and improves steps.

Change Headers to be Maps

Previously headers were lists of name/value tuples, e.g.:

[{"content-type", "text/html"}]

This is a standard across the ecosystem (with minor difference that some Erlang libraries use charlists instead of binaries.)

There are some problems with this particular choice though:

• We cannot use headers[name]
• We cannot use pattern matching

In short, this representation isn’t very ergonomic to use.

Now headers are maps of string names and lists of values, e.g.:

%{"content-type" => ["text/html"]}

This allows headers[name] usage:

response.headers["content-type"]
#=> ["text/html"]

and pattern matching:

case Req.request!(req) do
  %{headers: %{"content-type" => ["application/json" <> _]}} ->
    # handle JSON response
end

This is a major breaking change. If you cannot easily update your app or your dependencies, do:

# config/config.exs
config :req, legacy_headers_as_lists: true

This legacy fallback will be removed on Req 1.0.

There are two other changes to headers in this release.

Header names are now case-insensitive in functions like Req.Response.get_header/2.

Trailer headers, or more precisely trailer fields or simply trailers, are now stored in a separate trailers field on the %Req.Response{} struct as long as you use Finch 0.17+.

Add Request Body Streaming

Req v0.4 adds official support for request body streaming by setting the request body to an enumerable. Here’s an example:

iex> stream = Stream.duplicate("foo", 3)
iex> Req.post!("https://httpbin.org/post", body: stream).body["data"]
"foofoofoo"

The enumerable is passed through request steps and they may change it. For example, the compress_body step gzips the request body on the fly.

Add Response Body Streaming

Req v0.4 also adds response body streaming, via the :into option.

Here’s an example where we download the first 20kb (by making a range request, via the put_range step) of Elixir release zip. We stream the response body into a function and can handle each body chunk. The function receives a {:data, data}, {req, resp} and returns a {:cont | :halt, {req, resp}} tuple.

resp =
  Req.get!(
    url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
    range: 0..20_000,
    into: fn {:data, data}, {req, resp} ->
      IO.inspect(byte_size(data), label: :chunk)
      {:cont, {req, resp}}
    end
  )

# output: 17:07:38.131 [debug] redirecting to https://objects.githubusercontent.com/github-production-release-asset-2e6(...)
# output: chunk: 16384
# output: chunk: 3617

resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> ""

Notice we only stream response body, that is, Req automatically handles HTTP response status and headers. Once the stream is done, Req passes the response through response steps which allows following redirects, retrying on errors, etc. Response body is set to empty string "" which is then ignored by decompress_body, decode_body, and similar steps. If you need to decompress or decode incoming chunks, you need to do that in your custom into: fun function.

As the name :into implies, we can also stream response body into any Collectable. Here’s a similar snippet to above where we stream to a file:

resp =
  Req.get!(
    url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
    range: 0..20_000,
    into: File.stream!("elixit-otp-26.zip.1")
  )

# output: 17:07:38.131 [debug] redirecting to (...)
resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> %File.Stream{}

Full CHANGELOG

• Change request.headers and response.headers to be maps.

• Ensure request.headers and response.headers are downcased.

  Per RFC 9110: HTTP Semantics, HTTP headers should be case-insensitive. However, per RFC 9113: HTTP/2 headers must be sent downcased.

  Req headers are now stored internally downcased and all accessor functions like Req.Response.get_header/2 are downcasing the given header name.

• Add trailers field to Req.Response struct. Trailer field is only filled in on Finch 0.17+.

• Make request.registered_options internal representation private.

• Make request.options internal representation private.

  Currently request.options field is a map but it may change in the future. One possible future change is using keywords lists internally which would allow, for example, Req.new(params: [a: 1]) |> Req.update(params: [b: 2]) to keep duplicate :params in request.options which would then allow to decide the duplicate key semantics on a per-step basis. And so, for example, put_params would merge params but most steps would simply use the first value.

  To have some room for manoeuvre in the future we should stop pattern matching on request.options. Calling request.options[key], put_in(request.options[key], value), and update_in(request.options[key], fun) is allowed.

• Fix typespecs for some functions

• Deprecate output step in favour of into: File.stream!(path).

• Rename follow_redirects step to redirect

• redirect: Rename :follow_redirects option to :redirect.

• redirect: Rename :location_trusted option to :redirect_trusted.

• redirect: Change HTTP request method to GET only on POST requests that result in 301..303.

  Previously we were changing the method to GET for all 3xx except 307 and 308.

• decompress_body: Remove support for deflate compression (which was broken)

• decompress_body: Don’t crash on unknown codec

• decompress_body: Fix handling HEAD requests

• decompress_body: Re-calculate content-length header after decompresion

• decompress_body: Remove content-encoding header after decompression

• decode_body: Do not decode response with content-encoding header

• run_finch: Add :inet6 option

• retry: Support retry: :safe_transient which retries HTTP 408/429/500/502/503/504 or exceptions with reason field set to :timeout/:econnrefused.

  :safe_transient is the new default retry mode. (Previously we retried on 408/429/5xx and any exception.)

• retry: Support retry: :transient which is the same as :safe_transient except it retries on all HTTP methods

• retry: Use retry-after header value on HTTP 503 Service Unavailable. Previously only HTTP 429 Too Many Requests was using this header value.

• retry: Support retry: &fun/2. The function receives request, response_or_exception and returns either:

  • true - retry with the default delay

  • {:delay, milliseconds} - retry with the given delay

  • false/nil - don’t retry

• retry: Deprecate retry: :safe in favour of retry: :safe_transient

• retry: Deprecate retry: :never in favour of retry: false

• Req.request/2: Improve error message on invalid arguments

• Req.update/2: Do not duplicate headers

• Req.update/2: Merge :params

• Req.Request: Fix displaying redacted basic authentication

• Req.Request: Add Req.Request.get_option/3

• Req.Request: Add Req.Request.fetch_option/2

• Req.Request: Add Req.Request.fetch_option!/2

• Req.Request: Add Req.Request.delete_option/2

• Req.Response: Add Req.Response.delete_header/2

• Req.Response: Add Req.Response.update_private/4

Hey everyone, some Req updates since v0.4.0 below.

New step: checksum

Sets expected response body checksum.

iex> resp = Req.get!("https://httpbin.org/json", checksum: "sha1:9274ffd9cf273d4a008750f44540c4c5d4c8227c")
iex> resp.status
200

iex> Req.get!("https://httpbin.org/json", checksum: "sha1:bad")
** (Req.ChecksumMismatchError) checksum mismatch
expected: sha1:bad
actual:   sha1:9274ffd9cf273d4a008750f44540c4c5d4c8227c

New step put_aws_sigv4

Signs request with AWS Signature Version 4.

iex> req =
...>   Req.new(
...>     base_url: "https://s3.amazonaws.com",
...>     aws_sigv4: [
...>       access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
...>       secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
...>       service: :s3
...>     ]
...>   )
iex>
iex> %{status: 200} = Req.put!(req, "/bucket1/key1", body: "Hello, World!")
iex> resp = Req.get!(req, "/bucket1/key1").body
"Hello, World!"

Request body streaming also works (though content-length header must be explicitly set):

iex> path = "a.txt"
iex> File.write!(path, String.duplicate("a", 100_000))
iex> size = File.stat!(path).size
iex> chunk_size = 10 * 1024
iex> stream = File.stream!(path, chunk_size)
iex> %{status: 200} = Req.put!(req, url: "/key1", headers: [content_length: size], body: stream)
iex> byte_size(Req.get!(req, "/bucket1/key1").body)
100_000

Req.Test

Functions for creating test stubs.

Stubs provide canned answers to calls made during the test, usually not responding at all to
anything outside what’s programmed in for the test.

“Mocks Aren’t Stubs” by Martin Fowler

Req has built-in support for stubs via :plug, :adapter, and (indirectly) :base_url
options. This module enhances these capabilities by providing:

• Stub any value with Req.Test.stub(name, value) and access it with
  Req.Test.stub(name). These functions can be used in concurrent tests.

• Access plug stubs with plug: {Req.Test, name}.

• Easily create JSON responses for Plug stubs with Req.Test.json(conn, body).

Example

Imagine we’re building an app that displays weather for a given location using an HTTP weather
service:

defmodule MyApp.Weather do
  def get_rating(location) do
    case get_temperature(location) do
      {:ok, %{status: 200, body: %{"celsius" => celsius}}} ->
        cond do
          celsius < 18.0 -> {:ok, :too_cold}
          celsius < 30.0 -> {:ok, :nice}
          true -> {:ok, :too_hot}
        end

      _ ->
        :error
    end
  end

  def get_temperature(location) do
    [
      base_url: "https://weather-service"
    ]
    |> Keyword.merge(Application.get_env(:myapp, :weather_req_options, []))
    |> Req.request(options)
  end
end

We configure it for production:

# config/runtime.exs
config :myapp, weather_req_options: [
  auth: {:bearer, System.fetch_env!("MYAPP_WEATHER_API_KEY")}
]

And tests:

# config/test.exs
config :myapp, weather_req_options: [
  plug: {Req.Test, MyApp.Weather}
]

And now we can easily stub out values in concurrent tests:

use ExUnit.Case, async: true

test "nice weather" do
  Req.Test.stub(MyApp.Weather, fn conn ->
    Req.Test.json(conn, %{"celsius" => 25.0})
  end)

  assert MyApp.Weather.get_rating("Krakow, Poland") == {:ok, :nice}
end

Full changelog below:

v0.4.10 (2024-02-19)

• run_finch: Default to connect_options: [protocols: [:http1, :http2]].

• run_finch: Change version requirement to ~> 0.17, that is all versions up to 1.0.

• put_aws_sigv4: Support streaming request body.

• auth: Always update authorization header.

• decode_body: Gracefully handle multiple content-type values.

• Req.Request.new/1: Use URI.parse for now.

v0.4.9 (2024-02-14)

• retry: Raise on invalid return from :retry_delay function

• run_finch: Update to Finch 0.17

• run_finch: Deprecate connect_options: [protocol: ...] in favour of
  connect_options: [protocols: ...]] which defaults to [:http1, :http2], that is,
  make request using HTTP/1 but if negotiated switch to HTTP/2 over the HTTP/1 connection.

• New step: put_aws_sigv4 - signs request with AWS Signature Version 4.

v0.4.8 (2023-12-11)

• put_plug: Fix response streaming. Previously we were relying on unreleased
  Plug features (which may never get released). Now, Plug adapter will emit the
  entire response body as one chunk. Thus,
  plug: plug, into: fn ... -> {:halt, acc} end is not yet supported as it
  requires Plug changes that are still being discussed. On the flip side,
  we should have much more stable Plug integration regardless of this small
  limitation.

v0.4.7 (2023-12-11)

• put_plug: Don’t crash if plug is not installed and :plug is not used

v0.4.6 (2023-12-11)

• New step: checksum
• put_plug: Fix response streaming when plug uses send_resp or send_file
• retry: Retry on :closed

v0.4.5 (2023-10-27)

• decompress_body: Remove content-length header

• auth: Deprecate auth: {user, pass} in favour of auth: {:basic, "user:pass"}

• Req.Request: Allow steps to be {mod, fun, args}

v0.4.4 (2023-10-05)

• compressed: Check for optional depenedencies brotli and ezstd only at compile-time.
  (backported from v0.3.12.)

• decode_body: Check for optional depenedency nimble_csv at compile-time.
  (backported from v0.3.12.)

• run_finch: Add :finch_private option

v0.4.3 (2023-09-13)

• Req.new/1: Fix setting :redact_auth

• Req.Request: Add Req.Request.get_option_lazy/3

• Req.Request: Add Req.Request.drop_options/2

v0.4.2 (2023-09-04)

• put_plug: Handle response streaming on Plug 1.15+.

• Don’t warn on mixed-case header names

v0.4.1 (2023-09-01)

• Fix Req.Request Inspect regression

Hey everyone, Req v0.5 is out! I’ve written a release blog post for the occasion. Happy hacking!

Req v0.5.0 brings testing enhancements, errors standardization, %Req.Response.Async{}, and more improvements and bug fixes.

Testing Enhancements

In previous releases, we could only create test stubs (using Req.Test.stub/2), that is, fake
HTTP servers which had predefined behaviour. Let’s say we’re integrating with a third-party
weather service and we might create a stub for it like below:

Req.Test.stub(MyApp.Weather, fn conn ->
  Req.Test.json(conn, %{"celsius" => 25.0})
end)

Anytime we hit this fake we’ll get the same result. This works extremely well for simple
integrations however it’s not quite enough for more complicated ones. Imagine we’re using
something like AWS S3 and we test uploading some data and reading it back again. While we could do
this:

Req.Test.stub(MyApp.S3, fn
  conn when conn.method == "PUT" ->
    # ...

  conn when conn.method == "GET" ->
    # ...
end)

making the test just a little bit more thorough will make it MUCH more complicated, for example:
the first GET request should return a 404, we then make a PUT, and now GET should return a 200.
We could solve it by adding some state to our test (e.g. an agent) but there is a simpler way and
that is to set request expectations using the new Req.Test.expect/3 function:

Req.Test.expect(MyApp.S3, fn conn when conn.method == "GET" ->
  Plug.Conn.send_resp(conn, 404, "not found")
end)

Req.Test.expect(MyApp.S3, fn conn when conn.method == "PUT" ->
  {:ok, body, conn} = Plug.Conn.read_body(conn)
  assert body == "foo"
  Plug.Conn.send_resp(conn, 200, "")
end)

Req.Test.expect(MyApp.S3, fn conn when conn.method == "GET" ->
  Plug.Conn.send_resp(conn, 200, "foo")
end)

The important part is the request expectations are meant to run in order (and fail if they don’t).

In this release we’re also adding Req.Test.transport_error/2, a way to simulate network
errors.

Here is another example using both of the new features, let’s simulate a server that is
having issues: on the first request it is not responding and on the following two requests it
returns an HTTP 500. Only on the third request it returns an HTTP 200. Req by default
automatically retries transient errors (using retry step) so it will make multiple
requests exercising all of our request expectations:

iex> Req.Test.expect(MyApp.S3, &Req.Test.transport_error(&1, :econnrefused))
iex> Req.Test.expect(MyApp.S3, 2, &Plug.Conn.send_resp(&1, 500, "internal server error"))
iex> Req.Test.expect(MyApp.S3, &Plug.Conn.send_resp(&1, 200, "ok"))
iex> Req.get!(plug: {Req.Test, MyApp.S3}).body
# 15:57:06.309 [error] retry: got exception, will retry in 1000ms, 3 attempts left
# 15:57:06.309 [error] ** (Req.TransportError) connection refused
# 15:57:07.310 [error] retry: got response with status 500, will retry in 2000ms, 2 attempts left
# 15:57:09.311 [error] retry: got response with status 500, will retry in 4000ms, 1 attempt left
"ok"

Finally, for parity with Mox, we add functions for setting ownership
mode:

• Req.Test.set_req_test_from_context/1
• Req.Test.set_req_test_to_private/1
• Req.Test.set_req_test_to_shared/1

And for verifying expectations:

• Req.Test.verify!/0
• Req.Test.verify!/1
• Req.Test.verify_on_exit!/1

Thanks to Andrea Leopardi for driving the testing improvements.

Standardized Errors

In previous releases, when using the default adapter, Finch, Req could return these exceptions on
network/protocol errors: Mint.TransportError, Mint.HTTPError, and Finch.Error. They have
now been standardized into: Req.TransportError and Req.HTTPError for more consistent
experience. In fact, this standardization was the pre-requisite of adding
Req.Test.transport_error/2!

Two additional exception structs have been added: Req.ArchiveError and Req.DecompressError
for zip/tar/etc errors in decode_body and gzip/br/zstd/etc errors in decompress_body
respectively. Additionally, decode_body now returns Jason.DecodeError instead of raising it.

%Req.Response.Async{}

In previous releases we added ability to stream response body chunks into the current process
mailbox using the into: :self option. When such is used, the response.body is now set to
Req.Response.Async struct which implements the Enumerable protocol.

Here’s a quick example:

resp = Req.get!("http://httpbin.org/stream/2", into: :self)
resp.body
#=> #Req.Response.Async<...>
Enum.each(resp.body, &IO.puts/1)
# {"url": "http://httpbin.org/stream/2", ..., "id": 0}
# {"url": "http://httpbin.org/stream/2", ..., "id": 1}

Here is another example where we use Req to talk to two different servers. The first server
produces some test data, strings "foo", "bar" and "baz". The second one is an “echo” server, it simply
responds with the request body it returned. We then stream data from one server, transform it, and
stream it to the other one:

Mix.install([
  {:req, "~> 0.5"},
  {:bandit, "~> 1.0"}
])

{:ok, _} =
  Bandit.start_link(
    scheme: :http,
    port: 4000,
    plug: fn conn, _ ->
      conn = Plug.Conn.send_chunked(conn, 200)
      {:ok, conn} = Plug.Conn.chunk(conn, "foo")
      {:ok, conn} = Plug.Conn.chunk(conn, "bar")
      {:ok, conn} = Plug.Conn.chunk(conn, "baz")
      conn
    end
  )

{:ok, _} =
  Bandit.start_link(
    scheme: :http,
    port: 4001,
    plug: fn conn, _ ->
      {:ok, body, conn} = Plug.Conn.read_body(conn)
      Plug.Conn.send_resp(conn, 200, body)
    end
  )

resp = Req.get!("http://localhost:4000", into: :self)
stream = resp.body |> Stream.with_index() |> Stream.map(fn {data, idx} -> "[#{idx}]#{data}" end)
Req.put!("http://localhost:4001", body: stream).body
#=> "[0]foo[1]bar[2]baz"

Req.Response.Async is an experimental feature which may change in the future.

The existing caveats to into: :self still apply, that is:

• If the request is sent using HTTP/1, an extra process is spawned to consume messages from the
  underlying socket.

• On both HTTP/1 and HTTP/2 the messages are sent to the current process as soon as they arrive,
  as a firehose with no back-pressure.

If you wish to maximize request rate or have more control over how messages are streamed, use
into: fun or into: collectable instead.

Full v0.5.0 CHANGELOG

• Req: Deprecate setting :headers to values other than string/integer/DateTime.
  This is to potentially allow special handling of atom values in the future.

• Req: Add Req.run/2 and Req.run!/2.

• Req: into: :self now sets response.body as Req.Response.Async which implements
  enumerable.

• Req.Request: Deprecate setting :redact_auth. It now has no effect. Instead of allowing
  to opt out of, we give an idea what the secret was without revealing it fully:

  iex> Req.new(auth: {:basic, "foobar:baz"})
  %Req.Request{
    options: %{auth: {:basic, "foo*******"}},
    ...
  }
  
  iex> Req.new(headers: [authorization: "bearer foobarbaz"])
  %Req.Request{
    headers: %{"authorization" => ["bearer foo******"]},
    ...
  }
  

• Req.Request: Deprecate halt/1 in favour of Req.Request.halt/2.

• Req.Test: Add Req.Test.transport_error/2 to simulate transport errors.

• Req.Test: Add Req.Test.expect/3.

• Req.Test: Add functions for setting ownership mode: Req.Test.set_req_test_from_context/1, Req.Test.set_req_test_to_private/1,
  Req.Test.set_req_test_to_shared/1 and for verifying expectations: Req.Test.verify!/0, Req.Test.verify!/1, and Req.Test.verify_on_exit!/1.

• Req.Test: Add Req.Test.html/2.

• Req.Test: Add Req.Test.text/2.

• Req.Test: Drop :nimble_ownership dependency.

• Req.Test: Deprecate Req.Test.stub/1, i.e. the intended use case is to only work
  with plug stubs/mocks.

• decode_body: Return Jason.DecodeError on JSON errors instead of raising it.

• decode_body: Return Req.ArchiveError on tar/zip errors.

• decompress_body: Return Req.DecompressError.

• put_aws_sigv4: Drop :aws_signature dependency.

• retry: (BREAKING CHANGE) Consider
  %Req.TransportError{reason: :closed | :econnrefused | :timeout} as transient. Previously
  any exceptions with those reason values were consider as such.

• retry: (BREAKING CHANGE) Consider
  %Req.HTTPError{protocol: :http2, reason: :unprocessed} as transient.

• run_finch: (BREAKING CHANGE) Return Req.HTTPError instead of Mint.HTTPError.

• run_finch: (BREAKING CHANGE) Return Req.TransportError instead of Mint.TransportError.

• run_finch: Set inet6: true if URL looks like IPv6 address.

• put_plug: Move most documentation to run_plug.

• run_plug: Make public.

• run_plug: Add support for simulating network issues using Req.Test.transport_error/2.

• run_plug: Support passing 2-arity functions as plugs.

• run_plug: Automatically fetch query params.

• verify_checksum: Fix handling compressed responses.