HTTP client libraries and wrappers

Here are the list of HTTP client libraries/wrappers, and some thoughts on HTTP client in general. I’d like to hear from others how they work with HTTP…

HTTP client libraries

HTTP Client Wrappers

Related links

Thoughts & Questions

  • Although HTTP spec says headers are case-insensitive, http client libraries should not automatically downcase such values (especially for outgoing request) since there are applications require specific cases :frowning:
  • headers, form data, and query string should be a list not a map to preserve orders (both order of keys and order of values)
  • For performance, what about using NIF to parse headers? For example, puma (app server in Ruby) uses C for this: https://github.com/puma/puma/tree/v3.12.0/ext/puma_http11 - such parser may be shared across client/server.
  • How should HTTP libraries handle HTTP version upgrade?
  • Should HTTP libraries make pure functional (zero side effect) or leverage more global states (connection pools, keeping connection, etc.)?
    • For example, Tesla allows creating new client on the fly, so that it “builds” client without any configuration from “global” config; however as it may use HTTP client library (application) which maintains some state in it.
16 Likes

In the meta-sense of the question, yeah, I’m not loving the fragmentation in http client libraries, however, it’s not a huge detriment to my day to day experience. I’d need to see some specific use cases to get behind some of your arguments about case and ordering, but it’s not a big deal because none of those things you’re advocating would work against any use case I’ve dealt with.

In the end, are you expecting someone else to write this ultra http client library, or are you trying to determine whether to write it yourself?

1 Like

So far I’m happy with Tesla (with httpc and hackeny) for small traffic. However for future it would be nice if I can use one library for http2 and further so this is my part of research.

For ordering - some services require signature of headers so it is required to keep the order.

I was wondering if there was a library that leveraged libcurl, and there is! It’s actively maintained too. I’m going to give it a try later to see if it works.

3 Likes

Why not do some code generation and make use of Elixir’s pattern matching, like so?

  def header_pair("Content-Encoding" <> rest), do: {"Content-Encoding", header_value(rest)}
  def header_pair("If-None-Match" <> rest),    do: {"If-None-Match",    header_value(rest)}
  # ^ these can be generated with a macro

  def header_value(val) do
    String.split(val, ~r/\s*:\s*/, trim: true)
    |> hd
  end

I am sure C code will be faster, however crossing the VM <-> native barrier has an overhead as well. I have not measured the native approach but I would turn the question back to you: have you established with certainty that Elixir HTTP header parsing is a bottleneck in your workflow?

(EDIT: on a second thought, this is not practical if we want to accept all possible case combinations for the headers.)

BTW, awesome job compiling the list! :023: :041:

Good news and bad news for Katipo.

Bad news: It depends on a metrics library that is, as far as I can tell, out of date. It uses merl to dynamically build a metrics module and it’s not getting through erl_lint on OTP 21.

Good news: The metrics module can easily be faked so that there are no problems making requests.

Here’s the fake module:

defmodule :metrics_mod do
  def new(_name, _type, _config), do: :ok
  def update(_name, _probe, _config), do: :ok
  def update_or_create(_, _, _), do: :ok
  def update_or_create(_name, _probe, _type, _config), do: :ok
  def delete(_name, _config), do: :ok
end
1 Like

I haven’t work with NIF, and I’m not arguing it’s a bottleneck.

However, as it may be called so many times, so even small improvement may give significant improvement (like JSON) in some cases. I guess it depends at which boundary NIF is used - for example, using NIF to parse each HTTP header may not worth it. However, we may overage NIF to take a stream (or string) of a HTTP response, and let it returns the whole parsed results - maybe less memory copying?

From ElixirConf 2018 keynote - see https://youtu.be/suOzNeMJXl0?t=2207

4 Likes

Question here: in your opinion should client HTTP libs optionally deal with response caching, i.e. dealing with the cache-control, pragma, expires, etc. headers, and cache the response automatically? Or should it better be done by the application?
There’s for example that HTTPoison issue from 2 years ago. Not sure if it’d be a sane approach.

All libs should have “minimal” default behavior. I don’t think HTTP cache or auto-retry for idempotent requests should be turned on by default.

A good library is extensible (like Elixir), not full-featured. That’s why Tesla is my current choice of HTTP wrapper.

Note that httpc does not verify certificate by default

2 Likes

Neither do ibrowse, HTTPotion, gun, Tesla and SimpleHttp. And neither do hackney and HTTPoison if you pass any custom SSL options (e.g. select a custom CA, or suppress log messages with log_alert: false) without also passing verify: :verify_peer along with the right verify_fun

4 Likes

xhttp is renamed to mint

Mint is different from most Erlang and Elixir HTTP clients because it provides a process-less architecture. Instead, Mint is based on a functional and immutable data structure that represents an HTTP connection. This data structure wraps a TCP or SSL socket. This allows for more fine-tailored architectures where the developer is responsible for wrapping the connection struct, such as having one process handle multiple connections or having different kinds of processes handle connections.

5 Likes

Some time ago I started implement small client library use httpc. This library definitely don’t cover all needs, but I working on it. Recently was added the ability to pass the headers and body of the request as an argument. In any case, this library can be helpful shot - is a small HTTP client library for Erlang.

2 Likes

I don’t mean to degrade, just curious if you compared your library to the Gun Erlang HTTP client?

Sure, I know the gun library, but gun use tcp connection, sometime we don’t need to keep connection with some services and just need to call some REST API - what was implemented inside of the shot library

Not sure I understand. You mean you don’t want the HTTP Keep-Alive option used when calling a REST API?

No, I mean, the gun use tcp connection and some time we don’t need worries about it and we need just post/get something without knowing of socket or pid of connection. The shot is wrapper of httpc.
Eg: using of gun

{ok, ConnPid} = gun:open("example.org", 443),
StreamRef = gun:get(ConnPid, "/organizations/ninenines").

Eg: using of shot:

shot:get("http://example.org/somedata").

Oh, ergonomy and convenience. Thanks for clarifying.

1 Like
6 Likes

I ran into a strange issue today when connecting to an upstream server with Mojito - there’s a custom header which is being converted to lower case, however the upstream server expects the exact casing. Though I think this is a bad design, is there an option to force Mojito or Mint to maintain your case without converting them.