dlesl

Erqwest - A fast and correct HTTP client based on reqwest

Erqwest is an http client implemented as a NIF-based wrapper around reqwest using rustler. The aim is to deliver the best possible performance and correctness. It has reached the stage where it could do with some testing so if you would like to help with that, please try it out .

It’s written in erlang but it’s a simple API and should be ergonomic from elixir too:

iex(1)> :erqwest.start_client(:default)
:ok
iex(2)> {:ok, %{status: 200, body: body}} = :erqwest.get(:default, "https://httpbin.org/get")
{:ok,
 %{
   body: "{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Host\": \"httpbin.org\", \n    \"X-Amzn-Trace-Id\": \"Root=1-6108502f-5ff7a84e1e0c9843706ebc67\"\n  }, \n  \"origin\": \"85.230.179.80\", \n  \"url\": \"https://httpbin.org/get\"\n}\n",
   headers: [
     {"date", "Mon, 02 Aug 2021 20:06:07 GMT"},
     {"content-type", "application/json"},
     {"content-length", "221"},
     {"connection", "keep-alive"},
     {"server", "gunicorn/19.9.0"},
     {"access-control-allow-origin", "*"},
     {"access-control-allow-credentials", "true"}
   ],
   status: 200
 }}
iex(3)> :erqwest.req(:default, %{method: :put, url: "https://httpbin.org/delay/1", timeout: 100})
{:error,
 %{
   code: :timeout,
   reason: "error sending request for url (https://httpbin.org/delay/1): operation timed out"
 }}

It exposes most of the features commonly used in calling APIs etc, if there’s any reqwest feature that’s missing feel free to open an issue or PR!

14 comments

#http_client #library #rustler

46 3111 14

2023-05-21 01:09:52 UTC

Most Liked

dlesl

I’ve just released 0.1.0. It contains quite a few new features, the most noticeable being:

Making sure we’re a well-behaved NIF and always return in less than 1 ms. Previously this was not guaranteed to be the case when the request body was very large.
[Breaking change] Splitting the sync and async APIs for clarity. Everything in erqwest is now synchronous and the async interface lives in erqwest_async.
Streaming support (of request and response bodies). Getting the API right was a bit tricky but I think it worked out well. I wanted to ensure that the sync API is always “safe”, ie. that it is impossible to end up in a state where a function call hangs indefinitely waiting for a message, or where you end up with stray messages in your inbox, even if you use the API incorrectly. I also wanted to ensure that the ergonomics of non-streaming use are not compromised.
[Breaking change] As a result of the above the message format for the async API has changed.
[Breaking change] The tokio runtime is now monitored by an erlang supervisor. This means you need to start the application (:application.ensure_started(:erqwest)) before using it.
Optional cookies and gzip support. These are off by default since the extra dependencies increase the rust compile times. You set an env var at compile time to enable them (I’m not sure if there’s a better way to handle optional dependencies with rebar3, if you have ideas please let me know!).

See the readme for examples of how to use the new streaming API and grab the new version from hex where you can also find the docs

Post #5

dlesl

Sure, so the first thing I did was time the NIF calls, just using timer:tc/1, which I think is good enough here (if you run it a few times) because we’re not benchmarking, we just want a rough idea. You should look at the shortest times (assuming a CPU bound workload), since longer times are probably caused by the OS scheduler context switching. What I saw is that erqwest_nif:make_client/2 is consistently very slow (~30 ms). I checked what it was doing with perf and it was spending its time in openssl, which we can assume means it’s CPU bound, so I marked it as a CPU-bound dirty NIF.

For erqwest_nif:req/1, times were generally well under 1 ms, which makes sense because all it’s doing is queueing something to be processed by another thread. I went looking for any edge cases which might cause it to consistently take more than 1 ms, and found that it did when the request body was large. This is because it copies the body into a Vec<u8>. I considered marking this NIF as dirty too, however some benchmarking showed that this caused a ~30% slowdown in the (probably more common) case that the request body is small, so I looked for another solution. It turns out that copying a binary is very cheap, since binaries over 64-bytes are reference counted. So I changed the code to just copy the binary/iodata to an OwnedEnv, and decode it on a tokio thread where we have no execution time restrictions.

(I’m getting told my post has too many links so I’m splitting it up)

Post #7

wojtekmach

Hex Core Team

Really nice project and really convenient to use. I really like the decision to take e.g. headers as an option, and not an arg, and returning a response map.

I was curious to see how erqwest is using rustler in an Erlang project. And it turns out it does not use rustler, the Mix project, it just uses rustler crates. This is very neat, keeping dependencies to the absolute minimum. The only thing this library needs is rust & cargo.

Post #3