WestKeys

WestKeys

Mint vs Finch vs Gun vs Tesla vs HTTPoison etc

Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the tools to get behind, which results in very little fragmentation.

Surely someone here has experience with the majority of these and can save newcomers multiple hours of testing/fiddling around

This thread has some insight:

Non-exhaustive list of HTTP clients or arguably similar:

Other than ergonomics, how do you go about picking one?

Marked As Solved

keathley

keathley

Let me state up front that I’m the original author of Finch (although at this point I’m by no means the largest contributor to the project). So that’s my bias out of the way. I’ll attempt to give a rundown of the current clients and then explain why I felt like I should try to contribute a new one.

Hackney and httpoison (which is an elixir wrapper around hackney) are probably the oldest and most popular http clients for Elixir. Both are very battle tested and have been used in many production deployments of elixir. Hackney has a lot of features and supports a number of use cases. The main issue with hackney (and thus httpoison) is with how it handles pools of connections. The pools have slowly gotten better over time but hackney is attempting to support a large number of use cases and has defaults that are better in general but worse in high throughput scenarios. If you care about performance, you’ll need to create dedicated pools for each of your hosts, manage which pools you’re using for different calls, etc. This is all doable, but its a chore and error prone. hackney and httpoison don’t support :telemetry, at least last I checked, which also added friction. We used hackney for a very long time at B/R and got a very long way. I’m very appreciative of the work that people have put into those libraries.

Gun is less popular but claimed higher performance than hackney (using default configs) for a long time. That said, its always been a chore to get to work properly because it depends on a non-standard version of cowlib that makes it incompatible with stuff like plug. So getting it working in a standard elixir project is typically difficult. It also didn’t truly support all of the features (websockets comes to mind) that it claimed for a while. Some of this may have changed as I believe there was a new release recently.

Mint is a wrapper around gen_tcp and ssl libraries in erlang that allows you to make http/1.1 and http/2 requests. It does this in a way that mostly hides the underlying socket mechanisms from you. This is useful for building libraries but makes it highly non-ergonomic for general use. You’ll need to build your own pooling mechanism if you want to get performance benefits of long lived connections with http/1.1. http/2 is a highly stateful protocol so you’ll end up needing to implement a lot of logic and functionality on top of the connection to make it work correctly.

Mojito is a pooling solution written around Mint. It uses a novel pool of pools for specific downstreams which tends to provide good utilization of all your connections. The main downside to mojito is that each connection is stored inside of a single process. That means that when you need to make a request, you need to get access to the gen_server, send a request to the gen_server, the gen_server then receives the response, and returns it back to the caller. Whenever you cross a process boundary, the memory must be copied. This means that you’re doing excessive copying to and fro for every single request which increases memory pressure, increases CPU pressure due to excess garbage collection and process rescheduling, and increases your overall latency since the connection can’t be checked back into the pool until the copying has finished.

This brings me to finch. At B/R we were making tens of thousands of http requests per second and were running into errors with misuse of hackney pools. So I spent a bunch of my free time working on a new client that combined mint with a new pool that jose had recently published called nimble pool. Finch stole the idea of a pool of pools from mojito. But instead of using processes to hold each connection, we instead hand the connection itself to the caller (which is possible thanks to mint’s design). This reduces memory, cpu, and decreases latency since we’re no longer copying large binaries across process boundaries. At least, this is true for http/1.1. http/2 is a completely different implementation and I suspect is about as “fast” as any other http/2 implementation. Finch also added support for telemetry spans which was something that we needed at B/R and has made its way into other clients as well. But Finch is brutally focused on being high throughput. This means we don’t support as many features as other clients do because we would have to take them into consideration when dealing with performance goals.

A more pleasant to use API around Finch you should check out GitHub - wojtekmach/req. I think this is potentially the right way to handle the various features such as a REST verb API, automatically decompressing responses, retries, etc. that many people have come to expect from a robust client. Definitely watching and chatting with Wojtek about it and excited to see what he does with it.

Something else that I didn’t see mentioned in your original post is chatterbox. This is a http/2 only implementation for both clients and servers. If I was going to support only http/2 that would be what I would use atm.

Tesla is a wrapper around all of these things. I would stick to the hackney adapter or the finch adapter (because of my aforementioned biases). We attempted to use tesla at B/R for a long time but eventually found that it still required more configuration than we wanted to do per project and moved to GitHub - keathley/twirp-elixir: Elixir implementation of the twirp RPC framework, backed by finch. But Tesla has a lot of useful middleware and I think its a good library if you only have a few clients to manage.

I’ve never seen simplehttp or katipo so I can’t comment on either of those.

112
Post #11

Also Liked

voltone

voltone

Another thing to look at is whether a client handles TLS securely out-of-the-box. If it doesn’t, then you are going to have to do the ssl options dance.

Mint-based clients, including Finch, Mojito, and Tesla with the appropriate adapter, should handle that just fine.

Hackney, and by extension HTTPoison and Tesla with the Hackney adapter, do check the server certificate by default. However, if you want to start tuning the TLS options, e.g. to disable old TLS versions, you are going to have to set all the necessary options yourself:

# This fails, as expected:
iex(1)> HTTPoison.get("https://expired.badssl.com/")                                   
{:error,
 %HTTPoison.Error{
   id: nil,
   reason: {:tls_alert,
    {:certificate_expired,
     'TLS client: In state certify at ssl_handshake.erl:1764 generated CLIENT ALERT: Fatal - Certificate Expired\n'}}
 }}
# But this succeeds while it shouldn't:
iex(2)> HTTPoison.get("https://expired.badssl.com/", [], ssl: [versions: [:"tlsv1.2"]])
{:ok,
 %HTTPoison.Response{...}}

Other clients, such as httpc, gun and ibrowse, leave it completely up to you. For instance, Tesla with the default httpc adapter:

iex(3)> Tesla.get("https://expired.badssl.com/")                    
{:ok,
 %Tesla.Env{...}}

I would highly recommend you test your choice of HTTP client against some https://badssl.com endpoints, using the specific configuration (adapter, ssl options) you intend to use, so you can be confident your application gets the TLS protections you expect.

WestKeys

WestKeys

Thanks. I guess you guys interpreted this as “which library should a software noob pick”.
Many newcomers to Elixir are not newcomers to software. Here’s to hoping for a more thoughtful discussion.

15
Post #4
keathley

keathley

If you’re going to question how people spend their free time, you should at least offer to mow their lawn. I mean, it’s not surprising you’re getting downvoted. They’ve offered you code and insight into how they approach a problem - something that is much more valuable than the actual code btw. In return, you offered them an opinion on said code. When they didn’t respond to your opinion in the way you wanted, you criticized them. You’re being downvoted because you presumed that your opinion was worth more than that maintainers time. If you don’t like Tesla it’s really simple: don’t use Tesla. Or fork it and change it to your liking. But don’t assume that because you use a thing it gives you any right to tell people what they do with their projects or their time.

Where Next?

Popular in Questions Top

tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
earth10
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47849 226
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement