Here are the list of HTTP client libraries/wrappers, and some thoughts on HTTP client in general. I’d like to hear from others how they work with HTTP…
Although HTTP spec says headers are case-insensitive, http client libraries should not automatically downcase such values (especially for outgoing request) since there are applications require specific cases
headers, form data, and query string should be a list not a map to preserve orders (both order of keys and order of values)
How should HTTP libraries handle HTTP version upgrade?
Should HTTP libraries make pure functional (zero side effect) or leverage more global states (connection pools, keeping connection, etc.)?
For example, Tesla allows creating new client on the fly, so that it “builds” client without any configuration from “global” config; however as it may use HTTP client library (application) which maintains some state in it.
In the meta-sense of the question, yeah, I’m not loving the fragmentation in http client libraries, however, it’s not a huge detriment to my day to day experience. I’d need to see some specific use cases to get behind some of your arguments about case and ordering, but it’s not a big deal because none of those things you’re advocating would work against any use case I’ve dealt with.
In the end, are you expecting someone else to write this ultra http client library, or are you trying to determine whether to write it yourself?
So far I’m happy with Tesla (with httpc and hackeny) for small traffic. However for future it would be nice if I can use one library for http2 and further so this is my part of research.
For ordering - some services require signature of headers so it is required to keep the order.
I was wondering if there was a library that leveraged libcurl, and there is! It’s actively maintained too. I’m going to give it a try later to see if it works.
Why not do some code generation and make use of Elixir’s pattern matching, like so?
def header_pair("Content-Encoding" <> rest), do: {"Content-Encoding", header_value(rest)}
def header_pair("If-None-Match" <> rest), do: {"If-None-Match", header_value(rest)}
# ^ these can be generated with a macro
def header_value(val) do
String.split(val, ~r/\s*:\s*/, trim: true)
|> hd
end
I am sure C code will be faster, however crossing the VM ↔ native barrier has an overhead as well. I have not measured the native approach but I would turn the question back to you: have you established with certainty that Elixir HTTP header parsing is a bottleneck in your workflow?
(EDIT: on a second thought, this is not practical if we want to accept all possible case combinations for the headers.)
Bad news: It depends on a metrics library that is, as far as I can tell, out of date. It uses merl to dynamically build a metrics module and it’s not getting through erl_lint on OTP 21.
Good news: The metrics module can easily be faked so that there are no problems making requests.
I haven’t work with NIF, and I’m not arguing it’s a bottleneck.
However, as it may be called so many times, so even small improvement may give significant improvement (like JSON) in some cases. I guess it depends at which boundary NIF is used - for example, using NIF to parse each HTTP header may not worth it. However, we may overage NIF to take a stream (or string) of a HTTP response, and let it returns the whole parsed results - maybe less memory copying?
Question here: in your opinion should client HTTP libs optionally deal with response caching, i.e. dealing with the cache-control, pragma, expires, etc. headers, and cache the response automatically? Or should it better be done by the application?
There’s for example that HTTPoison issue from 2 years ago. Not sure if it’d be a sane approach.
Neither do ibrowse, HTTPotion, gun, Tesla and SimpleHttp. And neither do hackney and HTTPoison if you pass any custom SSL options (e.g. select a custom CA, or suppress log messages with log_alert: false) without also passing verify: :verify_peer along with the right verify_fun…
Mint is different from most Erlang and Elixir HTTP clients because it provides a process-less architecture. Instead, Mint is based on a functional and immutable data structure that represents an HTTP connection. This data structure wraps a TCP or SSL socket. This allows for more fine-tailored architectures where the developer is responsible for wrapping the connection struct, such as having one process handle multiple connections or having different kinds of processes handle connections.
Some time ago I started implement small client library use httpc. This library definitely don’t cover all needs, but I working on it. Recently was added the ability to pass the headers and body of the request as an argument. In any case, this library can be helpful shot - is a small HTTP client library for Erlang.
Sure, I know the gun library, but gun use tcp connection, sometime we don’t need to keep connection with some services and just need to call some REST API - what was implemented inside of the shot library
No, I mean, the gun use tcp connection and some time we don’t need worries about it and we need just post/get something without knowing of socket or pid of connection. The shot is wrapper of httpc.
Eg: using of gun
I ran into a strange issue today when connecting to an upstream server with Mojito - there’s a custom header which is being converted to lower case, however the upstream server expects the exact casing. Though I think this is a bad design, is there an option to force Mojito or Mint to maintain your case without converting them.
