wojtekmach
Hey everyone!
Req is an HTTP client for Elixir that I’ve been working on for quite some time. There is already a lot of HTTP clients out there so why create a new one? Two things: great out of the box experience and extensibility.
Regarding out of the box experience, let’s first see it in action:
Mix.install([
{:req, "~> 0.3.0"}
])
Req.get!("https://api.github.com/repos/elixir-lang/elixir").body["description"]
#=> "Elixir is a dynamic, functional language designed for building scalable and maintainable applications"
Req.get!("http://api.github.com").status
# 23:24:11.670 [debug] follow_redirects: redirecting to https://api.github.com/
#=> 200
Req.get!("https://httpbin.org/status/500,200").status
# 19:02:08.463 [error] retry: got response with status 500, will retry in 2000ms, 2 attempts left
# 19:02:10.710 [error] retry: got response with status 500, will retry in 4000ms, 1 attempt left
#=> 200
Req automatically decompress and decodes response body, follows redirects, retries in face of errors, and more. See “Features” section in the README for the whole list.
Regarding extensibility, virtually all of Req functionality is broken down into individual pieces - steps. Req works by running the request struct through these steps. You can easily reuse or rearrange built-in steps or write new ones. Steps are similar to Tesla Middleware although they are very different in implementation. Steps are just regular functions:
debug_url = fn request ->
IO.inspect(URI.to_string(request.url))
request
end
req =
Req.new(base_url: "https://api.github.com")
|> Req.Request.append_request_steps(debug_url: debug_url)
Req.get!(req, url: "/repos/wojtekmach/req").body["description"]
# Outputs: "https://api.github.com/repos/wojtekmach/req"
#=> "Req is a batteries-included HTTP client for Elixir."
See Req.Steps module for a list of all built-in steps.
After writing custom Req steps we can make them even easier to use by others by packaging them up into plugins. Here are some examples:
Mix.install([
{:req, "~> 0.3.0"},
{:req_easyhtml, github: "wojtekmach/req_easyhtml"},
{:req_s3, github: "wojtekmach/req_s3"},
{:req_hex, github: "wojtekmach/req_hex"}
])
req =
(Req.new(http_errors: :raise)
|> ReqEasyHTML.attach()
|> ReqS3.attach()
|> ReqHex.attach())
Req.get!(req, url: "https://elixir-lang.org").body[".entry-summary h5"]
#=>
# #EasyHTML[<h5>
# Elixir is a dynamic, functional language for building scalable and maintainable applications.
# </h5>]
Req.get!(req, url: "s3://ossci-datasets").body
#=>
# [
# "mnist/",
# "mnist/t10k-images-idx3-ubyte.gz",
# "mnist/t10k-labels-idx1-ubyte.gz",
# "mnist/train-images-idx3-ubyte.gz",
# "mnist/train-labels-idx1-ubyte.gz"
# ]
Req.get!(req, url: "https://repo.hex.pm/tarballs/req-0.1.0.tar").body["metadata.config"]["links"]
#=> %{"GitHub" => "https://github.com/wojtekmach/req"}
Plugins are nothing more than a convention (there’s no plugin contract) and I’m still figuring out what makes and doesn’t make sense to be a plugin. See “Writing Plugins” section in Req.Request module documentation for a little bit more information about plugins.
If you’re new to Req, I hope this post serves as a good introduction. If you have heard about it before, you may want to check the latest v0.3 release.
Any feedback is appreciated. Happy hacking!
Where Next?
Trending in Announcing
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
tj0
All praise sensible defaults! I got bitten once by one of the clients not verifying SSL. Now it’s the first thing I check.
sergio
Will swap out my Tesla usage with this. I love sensible defaults and Req seems less code to write. I hate writing code.
stefanchrobot
In my app I’m making requests to user-defined URLs, so errors are expected and I guess my feedback is mostly around that. Have a look at this:
Req.get(not only the!versions),And two general comments:
Req.request(foo: [bar: 1])meant that[bar: 1]are the options for the:foostep.wojtekmach
Thanks for feedback!
I might ultimately do that but no plans at the moment. I think the ! version is what most people would use most of the time, i.e. crash on transient errors (after N retries) or some serious misconfiguration (of eg ssl options), at which point it’s not like at the call site we can do much in that situation anyway so might as well crash. I’ll definitely keep this in mind though.
Do you have specific kinds of errors you want to differentiate between?
On the flip side if we’re erroring because service is under load, retrying again in a few milliseconds won’t help and in fact will make things worse. I went with the backoff values (1s, 2s, 4s, 8s, …) that curl has. It is very easy to provide your own retry strategy, fwiw.
Agreed. Looking at it again, retrying on transport errors (which, to your point, are adapter specific) needs rethinking. curl handles some of 4xx/5xx by default plus opts-in to handling ECONNREFUSED with
--retry-connrefusedand that seems sensible enough. I will look into that.Not sure what you mean by this, can you elaborate?
They don’t relate, it is arbitrary. I actually started with having options be named after steps and nesting but found it a bit too verbose at times, e.g.:
follow_redirects: [max_redirects: 10]. Besides this, there are options which are affecting multiple steps, for exampleraw: truewould disable both decompress_body and decode_body. Overall I’m pretty happy with the “flatter” options but totally understand the concern. Curious what others think.stefanchrobot
Not sure really, I just don’t like the fact that it’s an opaque term. That’s an issue that I have with a lot of HTTP clients and I’d like to see an improvement in this area. Can I log the error? It might contain sensitive information. I could pattern match, but then what happens when I switch adapters? Is it always going to be a map with a
:reasonkey?Makes sense. From my experience the retries usually address random network errors, so short retries are better. But I guess there’s no universal answer to this.
Would be nice nice to be able to just say
append_stepsinstead ofappend_*_stepssince I guess most of the time the step already implies where it should be.Unless I’d learn the steps and all the config options more or less by heart, the fact that they are arbitrary makes it confusing. I don’t like when things are arbitrary or too implicit. I think there are ways to address the verbosity - the step name implies what the settings do, so you could accept all of the following:
follow_redirects: [max: 10],follow_redirects: true,follow_redirects: 10.As for options that affects multiple steps, I’m not sure that’s the optimal design. Can I decompress the body without decoding it?
hubertlepicki
This is very nice. How hard would it be, in your opinion, to add support for the streaming request body and also streaming responses? I have a pretty specific use case where I generate files I need to send out on the fly, and I don’t want to store them at all, but they can be fairly large (> 1GB) so currently, I am generating Elixir Stream and feed it as request body to Finch (added that part to the lib) Finch — Finch v0.23.0 and when I receive the files on the other end I also turn them into Elixir stream and consume line by line. Unfortunately HTTP is all I have to talk to between both apps, but so far it’s been working great with Finch.
wojtekmach
What do you mean by opaque term? I believe it is pretty concrete. For Req by default it’s Finch.Error, Mint.HTTPError, or Mint.TransportError. You’re totally right that if switching adapters you’d get different errors so if you have error handling code, it would need to be updated. But what I want to figure out is what kind of error handling do you actually have. Because in my experience if I get any of these errors I cannot do anything with them anyway, I cannot recover from them, so the sensible thing is just to crash.
Perhaps I should make a bit less emphasis on being able to switch adapters because honestly I don’t see the point. Finch is great.
To me switching adapters is only useful in tests.
Gotcha, sorry, that is not possible. We have three buckets, request, response or error steps, so
when we add something we need to know what type of a thing it is, we cannot infer it.
I’m not sure if it addresses your concern but fwiw all the available options for built-in steps
are documented in a single place:
Req.request/1options.Yes, you can set
decode_body: false.wojtekmach
Thanks! Streaming request body is trivial as finch already does it (thank you for adding it!) but streaming response is pretty tricky. See Response streaming · Issue #82 · wojtekmach/req · GitHub for some discussion. It’s definitely on my mind and unless we have it, I won’t consider Req complete, but there’s no concrete plans at the moment, unfortunately.
stefanchrobot
You know the errors because you’re the author. As a user, I can only see this:
Not as opaque as hackney:
but still not very useful. Ideally this would be
{:error, Req.Error.t()}with a well-defined set of possible values.In my use case those errors are expected (e.g. mistyped URL), so I don’t want to crash.
Supporting only one adapter is perfectly fine for me - I was pretty happy with HTTPoison at some point. Support for replacing the adapter for tests is very important though.
wojtekmach
I’m skeptical about adding a Req.Error because I think it would be inferior to e.g. Mint.TransportError (it has a nice Exception.message/1 callback implementation) and I’m not sure I can reliably keep them in sync. I’m skeptical about adding strict error contract, one that would be useful for control flow, because I don’t think errors should be used for control flow. (I’m kind of doing that in retry step and maybe that’s my mistake.) I don’t have an answer for it but I’ll keep at it. Thanks for bringing this up!