Req — A batteries-included HTTP client for Elixir

[arton]

Has anyone written any articles on testing with Req yet? It seems like replacing the adapter with a mocked adapter is straightforward as described here, but I’m just wondering if there are any good examples of what people have been doing. I’m currently interested in replacing httpoison with Req on a project that uses Mox.

[wojtekmach]

Req v0.4 is out!

Req v0.4.0 changes headers to be maps, adds request & response streaming, and improves steps.

Change Headers to be Maps

Previously headers were lists of name/value tuples, e.g.:

[{"content-type", "text/html"}]

This is a standard across the ecosystem (with minor difference that some Erlang libraries use charlists instead of binaries.)

There are some problems with this particular choice though:

• We cannot use headers[name]
• We cannot use pattern matching

In short, this representation isn’t very ergonomic to use.

Now headers are maps of string names and lists of values, e.g.:

%{"content-type" => ["text/html"]}

This allows headers[name] usage:

response.headers["content-type"]
#=> ["text/html"]

and pattern matching:

case Req.request!(req) do
  %{headers: %{"content-type" => ["application/json" <> _]}} ->
    # handle JSON response
end

This is a major breaking change. If you cannot easily update your app or your dependencies, do:

# config/config.exs
config :req, legacy_headers_as_lists: true

This legacy fallback will be removed on Req 1.0.

There are two other changes to headers in this release.

Header names are now case-insensitive in functions like Req.Response.get_header/2.

Trailer headers, or more precisely trailer fields or simply trailers, are now stored in a separate trailers field on the %Req.Response{} struct as long as you use Finch 0.17+.

Add Request Body Streaming

Req v0.4 adds official support for request body streaming by setting the request body to an enumerable. Here’s an example:

iex> stream = Stream.duplicate("foo", 3)
iex> Req.post!("https://httpbin.org/post", body: stream).body["data"]
"foofoofoo"

The enumerable is passed through request steps and they may change it. For example, the compress_body step gzips the request body on the fly.

Add Response Body Streaming

Req v0.4 also adds response body streaming, via the :into option.

Here’s an example where we download the first 20kb (by making a range request, via the put_range step) of Elixir release zip. We stream the response body into a function and can handle each body chunk. The function receives a {:data, data}, {req, resp} and returns a {:cont | :halt, {req, resp}} tuple.

resp =
  Req.get!(
    url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
    range: 0..20_000,
    into: fn {:data, data}, {req, resp} ->
      IO.inspect(byte_size(data), label: :chunk)
      {:cont, {req, resp}}
    end
  )

# output: 17:07:38.131 [debug] redirecting to https://objects.githubusercontent.com/github-production-release-asset-2e6(...)
# output: chunk: 16384
# output: chunk: 3617

resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> ""

Notice we only stream response body, that is, Req automatically handles HTTP response status and headers. Once the stream is done, Req passes the response through response steps which allows following redirects, retrying on errors, etc. Response body is set to empty string "" which is then ignored by decompress_body, decode_body, and similar steps. If you need to decompress or decode incoming chunks, you need to do that in your custom into: fun function.

As the name :into implies, we can also stream response body into any Collectable. Here’s a similar snippet to above where we stream to a file:

resp =
  Req.get!(
    url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
    range: 0..20_000,
    into: File.stream!("elixit-otp-26.zip.1")
  )

# output: 17:07:38.131 [debug] redirecting to (...)
resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> %File.Stream{}

Full CHANGELOG

• Change request.headers and response.headers to be maps.

• Ensure request.headers and response.headers are downcased.

  Per RFC 9110: HTTP Semantics, HTTP headers should be case-insensitive. However, per RFC 9113: HTTP/2 headers must be sent downcased.

  Req headers are now stored internally downcased and all accessor functions like Req.Response.get_header/2 are downcasing the given header name.

• Add trailers field to Req.Response struct. Trailer field is only filled in on Finch 0.17+.

• Make request.registered_options internal representation private.

• Make request.options internal representation private.

  Currently request.options field is a map but it may change in the future. One possible future change is using keywords lists internally which would allow, for example, Req.new(params: [a: 1]) |> Req.update(params: [b: 2]) to keep duplicate :params in request.options which would then allow to decide the duplicate key semantics on a per-step basis. And so, for example, put_params would merge params but most steps would simply use the first value.

  To have some room for manoeuvre in the future we should stop pattern matching on request.options. Calling request.options[key], put_in(request.options[key], value), and update_in(request.options[key], fun) is allowed.

• Fix typespecs for some functions

• Deprecate output step in favour of into: File.stream!(path).

• Rename follow_redirects step to redirect

• redirect: Rename :follow_redirects option to :redirect.

• redirect: Rename :location_trusted option to :redirect_trusted.

• redirect: Change HTTP request method to GET only on POST requests that result in 301…303.

  Previously we were changing the method to GET for all 3xx except 307 and 308.

• decompress_body: Remove support for deflate compression (which was broken)

• decompress_body: Don’t crash on unknown codec

• decompress_body: Fix handling HEAD requests

• decompress_body: Re-calculate content-length header after decompresion

• decompress_body: Remove content-encoding header after decompression

• decode_body: Do not decode response with content-encoding header

• run_finch: Add :inet6 option

• retry: Support retry: :safe_transient which retries HTTP 408/429/500/502/503/504 or exceptions with reason field set to :timeout/:econnrefused.

  :safe_transient is the new default retry mode. (Previously we retried on 408/429/5xx and any exception.)

• retry: Support retry: :transient which is the same as :safe_transient except it retries on all HTTP methods

• retry: Use retry-after header value on HTTP 503 Service Unavailable. Previously only HTTP 429 Too Many Requests was using this header value.

• retry: Support retry: &fun/2. The function receives request, response_or_exception and returns either:

  • true - retry with the default delay

  • {:delay, milliseconds} - retry with the given delay

  • false/nil - don’t retry

• retry: Deprecate retry: :safe in favour of retry: :safe_transient

• retry: Deprecate retry: :never in favour of retry: false

• Req.request/2: Improve error message on invalid arguments

• Req.update/2: Do not duplicate headers

• Req.update/2: Merge :params

• Req.Request: Fix displaying redacted basic authentication

• Req.Request: Add Req.Request.get_option/3

• Req.Request: Add Req.Request.fetch_option/2

• Req.Request: Add Req.Request.fetch_option!/2

• Req.Request: Add Req.Request.delete_option/2

• Req.Response: Add Req.Response.delete_header/2

• Req.Response: Add Req.Response.update_private/4

[zambal]

Congrats with the release. I currently use httpoison for most of my http client needs, but will definitely consider Req the next time I need a http client.

Now that Req uses maps to store headers, I was wondering how Req handles multiple response headers with the same key. Fold them to a single entry with a list of values?

[LostKobrakai]

The map still has a list per key.

[MatijaL]

Hi, when using Req.get() with invalid URL, the process crashes and I get an error
scheme is required for url: and then there are some Finch functions mentioned. I can’t even pattern match on the error and return an error message back to the user.

[error] GenServer #PID<0.939.0> terminating
** (ArgumentError) scheme is required for url: ddsdsdf.com
    (finch 0.17.0) lib/finch/request.ex:135: Finch.Request.parse_url/1
    (finch 0.17.0) lib/finch/request.ex:103: Finch.Request.build/5
    (req 0.4.9) lib/req/steps.ex:753: Req.Steps.run_finch/1
    (req 0.4.9) lib/req/request.ex:993: Req.Request.run_request/1
    (req 0.4.9) lib/req/request.ex:938: Req.Request.run/1

Is there a way to solve this?

[LostKobrakai]

@MatijaL

Is there a way to solve this?

Don’t depend on Req doing the uri validation, but do it on your own. Req takes URI structs and that module has APIs for you to validate the input.

[stevensonmt]

I wonder if this should be recommended as a change in behavior for Finch. Currently Finch does this:

@doc false
  def parse_url(url) when is_binary(url) do
    url |> URI.parse() |> parse_url()
  end

  def parse_url(%URI{} = parsed_uri) do
     ...

but the URI.parse/1 docs point out

this function expects both absolute and relative URIs to be well-formed and does not perform any validation. See the “Examples” section below. Use new/1 if you want to validate the URI fields after parsing.

and

In contrast to URI.new/1, this function will parse poorly-formed URIs

Perhaps Finch should use URI.new/1 to parse the URL binary. I also think the guard for Finch.Request.parse_url/1 is not needed since both URI.new/1 and URI.parse/1 would handle both binaries and URI structs.

I also think the URI module has made an odd choice in having the parse function return uncommented success even when a string input cannot be a valid URI.

[LostKobrakai]

Afaik that would require increasing the minimum elixir version, which might be undesireable.

[wojtekmach]

@stevensonmt

Perhaps Finch should use URI.new/1 to parse the URL binary

This wouldn’t solve the problem above though:

iex> URI.new("ddsdsdf.com")
{:ok,
 %URI{
   scheme: nil,
   userinfo: nil,
   host: nil,
   port: nil,
   path: "ddsdsdf.com",
   query: nil,
   fragment: nil
 }}

[stevensonmt]

Weird that it considers that to be a “valid” URI. In that case maybe Finch could pattern match URI.parse against %URI{scheme: nil} to handle this situation?

[wojtekmach]

That’s basically what it does already!

sneako/finch/blob/v0.18.0/lib/finch/request.ex#L134:L135

[stevensonmt]

Oh yeah. My bad! Still feels like something not quite right about it though. Seems that a lib focused on explicitly handling web urls should maybe default to http or https rather than raising an error for a missing scheme. ddsdsdf.com should be recognized as an attempt at web url, shouldn’t it? We’re all used to just skipping the http:// part of any address we type into a browser.

[wojtekmach]

@stevensonmt

We’re all used to just skipping the http:// part of any address we type into a browser.

I haven’t seen any other HTTP client (besides good old curl) doing that. Please let me know if you have any such examples.

[wojtekmach]

Hey everyone, some Req updates since v0.4.0 below.

New step: checksum

Sets expected response body checksum.

iex> resp = Req.get!("https://httpbin.org/json", checksum: "sha1:9274ffd9cf273d4a008750f44540c4c5d4c8227c")
iex> resp.status
200

iex> Req.get!("https://httpbin.org/json", checksum: "sha1:bad")
** (Req.ChecksumMismatchError) checksum mismatch
expected: sha1:bad
actual:   sha1:9274ffd9cf273d4a008750f44540c4c5d4c8227c

New step put_aws_sigv4

Signs request with AWS Signature Version 4.

iex> req =
...>   Req.new(
...>     base_url: "https://s3.amazonaws.com",
...>     aws_sigv4: [
...>       access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
...>       secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
...>       service: :s3
...>     ]
...>   )
iex>
iex> %{status: 200} = Req.put!(req, "/bucket1/key1", body: "Hello, World!")
iex> resp = Req.get!(req, "/bucket1/key1").body
"Hello, World!"

Request body streaming also works (though content-length header must be explicitly set):

iex> path = "a.txt"
iex> File.write!(path, String.duplicate("a", 100_000))
iex> size = File.stat!(path).size
iex> chunk_size = 10 * 1024
iex> stream = File.stream!(path, chunk_size)
iex> %{status: 200} = Req.put!(req, url: "/key1", headers: [content_length: size], body: stream)
iex> byte_size(Req.get!(req, "/bucket1/key1").body)
100_000

Req.Test

Functions for creating test stubs.

Stubs provide canned answers to calls made during the test, usually not responding at all to
anything outside what’s programmed in for the test.

“Mocks Aren’t Stubs” by Martin Fowler

Req has built-in support for stubs via :plug, :adapter, and (indirectly) :base_url
options. This module enhances these capabilities by providing:

• Stub any value with Req.Test.stub(name, value) and access it with
  Req.Test.stub(name). These functions can be used in concurrent tests.

• Access plug stubs with plug: {Req.Test, name}.

• Easily create JSON responses for Plug stubs with Req.Test.json(conn, body).

Example

Imagine we’re building an app that displays weather for a given location using an HTTP weather
service:

defmodule MyApp.Weather do
  def get_rating(location) do
    case get_temperature(location) do
      {:ok, %{status: 200, body: %{"celsius" => celsius}}} ->
        cond do
          celsius < 18.0 -> {:ok, :too_cold}
          celsius < 30.0 -> {:ok, :nice}
          true -> {:ok, :too_hot}
        end

      _ ->
        :error
    end
  end

  def get_temperature(location) do
    [
      base_url: "https://weather-service"
    ]
    |> Keyword.merge(Application.get_env(:myapp, :weather_req_options, []))
    |> Req.request(options)
  end
end

We configure it for production:

# config/runtime.exs
config :myapp, weather_req_options: [
  auth: {:bearer, System.fetch_env!("MYAPP_WEATHER_API_KEY")}
]

And tests:

# config/test.exs
config :myapp, weather_req_options: [
  plug: {Req.Test, MyApp.Weather}
]

And now we can easily stub out values in concurrent tests:

use ExUnit.Case, async: true

test "nice weather" do
  Req.Test.stub(MyApp.Weather, fn conn ->
    Req.Test.json(conn, %{"celsius" => 25.0})
  end)

  assert MyApp.Weather.get_rating("Krakow, Poland") == {:ok, :nice}
end

Full changelog below:

v0.4.10 (2024-02-19)

• run_finch: Default to connect_options: [protocols: [:http1, :http2]].

• run_finch: Change version requirement to ~> 0.17, that is all versions up to 1.0.

• put_aws_sigv4: Support streaming request body.

• auth: Always update authorization header.

• decode_body: Gracefully handle multiple content-type values.

• Req.Request.new/1: Use URI.parse for now.

v0.4.9 (2024-02-14)

• retry: Raise on invalid return from :retry_delay function

• run_finch: Update to Finch 0.17

• run_finch: Deprecate connect_options: [protocol: ...] in favour of
  connect_options: [protocols: ...]] which defaults to [:http1, :http2], that is,
  make request using HTTP/1 but if negotiated switch to HTTP/2 over the HTTP/1 connection.

• New step: put_aws_sigv4 - signs request with AWS Signature Version 4.

v0.4.8 (2023-12-11)

• put_plug: Fix response streaming. Previously we were relying on unreleased
  Plug features (which may never get released). Now, Plug adapter will emit the
  entire response body as one chunk. Thus,
  plug: plug, into: fn ... -> {:halt, acc} end is not yet supported as it
  requires Plug changes that are still being discussed. On the flip side,
  we should have much more stable Plug integration regardless of this small
  limitation.

v0.4.7 (2023-12-11)

• put_plug: Don’t crash if plug is not installed and :plug is not used

v0.4.6 (2023-12-11)

• New step: checksum
• put_plug: Fix response streaming when plug uses send_resp or send_file
• retry: Retry on :closed

v0.4.5 (2023-10-27)

• decompress_body: Remove content-length header

• auth: Deprecate auth: {user, pass} in favour of auth: {:basic, "user:pass"}

• Req.Request: Allow steps to be {mod, fun, args}

v0.4.4 (2023-10-05)

• compressed: Check for optional depenedencies brotli and ezstd only at compile-time.
  (backported from v0.3.12.)

• decode_body: Check for optional depenedency nimble_csv at compile-time.
  (backported from v0.3.12.)

• run_finch: Add :finch_private option

v0.4.3 (2023-09-13)

• Req.new/1: Fix setting :redact_auth

• Req.Request: Add Req.Request.get_option_lazy/3

• Req.Request: Add Req.Request.drop_options/2

v0.4.2 (2023-09-04)

• put_plug: Handle response streaming on Plug 1.15+.

• Don’t warn on mixed-case header names

v0.4.1 (2023-09-01)

• Fix Req.Request Inspect regression

[sergio]

This release looks tremendous, appreciate all your hard work!

[margarza]

Hello! I am fairly new to the elixir language, so please bear with me

I have a question about the Req.Test module:

We are trying to test a method in case an API call fails with an status code of 404 and 500.

We have tried it with this code:

Req.Test.stub(:stub_req_plug, fn conn ->
      conn
      |> Plug.Conn.resp(404, "Not found")
      |> Plug.Conn.send_resp()
    end)

this returns {:ok, "Not Found"}, but we were wondering what we can do to get a return value of {:error, reason}

[zachallaun]

I believe you’re looking for the handle_http_errors step. It should look something like:

Req.get(test_endpoint, http_errors: :raise)

When you use the non-! variants (get as opposed to get!), this should result in the error tuple you’re looking for. When you use the ! variant, it’ll raise the exception.

[wojtekmach]

@zachallaun

this should result in the error tuple you’re looking for. When you use the ! variant, it’ll raise the exception.

http_errors: :raise always raises the exception. There were previous discussions of adding a non-raising variant that folks can find in the issue tracker.

@margarza

this returns {:ok, "Not Found"}, but we were wondering what we can do to get a return value of {:error, reason}

This is not possible using the :plug option at the moment. My advice would be to start a real HTTP server using bypass v2.1.0 — Documentation and simulate network failures using it. For example, you can set a low Req :receive_timeout and Process.sleep(1000) in your Bypass stub. Instead of setting plug: ... you’d be setting url: ... (usually base_url: ...). You can still use Req.Test module to stub out the url!

# config/test.exs
config :myapp, req_options: [
  # you need latest Req for :base_url to accept a function
  base_url: fn -> Req.Test.stub(:mystub) end
]

# some_test.exs
bypass = Bypass.open()
Req.Test.stub(:mystub, "http://localhost:#{bypass.port}")

Bypass.stub(bypass, fn conn ->
  ...
end)

The easiest network failure to test for is :econnrefused which you can get by hitting a port that is not bound so for example:

Req.Test.stub(:mystub, "http://localhost:9999")

assert MyApp.some_request() ==
         {:error, %Mint.TransportError{reason: :econnrefused}

Just a friendly reminder, if you are able to pass Req options down through all the layers, that’s preferred than going through Req.Test. For example:

- Req.Test.stub(:mystub, "http://localhost:9999")
- 
- assert MyApp.some_request() ==
+ assert MyApp.some_request(req_options: [base_url: "http://localhost:9999"]) ==
           {:error, %Mint.TransportError{reason: :econnrefused}

In future version of Req we plan to support simulating transport errors in :plug, something like this:

plug = fn _conn ->
  :econnrefused
end

assert Req.get(plug: plug) ==
         %Req.TransportError{reason: :econnrefused}

exact API TBD. Stay tuned!

[lud]

Hey @wojtekmach

We are using the new test functionality and it works great. But we want to run a couple integration tests and we would like to disable the test plug for those tests.

Is there a simple way to ignore the specified plug? We do not control the Req instance in the test, so we do not want to pass that option. We want to avoid messing with Application.put_env in the test if possible. We would like to set a flag in the test process and have the plug “ignore itself” if possible.

I see that if a plug is defined, the adapter will be set to run_plug/1, so our plug will always be called. Is there a safe way to define a custom plug and fallback to Req.Test by default, but call run_finch if the flag is set ?

(everything happens in the same process as the test)

[wojtekmach]

Req.Test.stub/1,2 can be used to stub anything so you could stub Req options:

defmodule MyApp.MyClient do
  def new(options) do
    Req.new(...)
    |> merge_test_options()
  end

  if Mix.env() == :test do
    defp merge_test_options(req) do
      Req.merge(req, plug: Req.Test.stub(__MODULE__))
    end
  else
    defp merge_test_options(req) do
      req
    end
  end
end

# test with stub:
Req.Test.stub(MyApp.MyClient, fn conn -> ... end)

# integration test:
Req.Test.stub(MyApp.MyClient, nil)

When you set plug: nil it is as if you never set the option in the first place.

It’s a little bit ugly to change production code to accomodate tests like but I believe it is the most straighforward option. There could be more indirect ways I did not consider yet.