Reliable HTTP libraries? (HTTPoison, HTTPotion, Tesla)?

Chat / Discussions
1

There was a post made quite a while back regarding the first two, which made mention of Tesla:

I originally started using HTTPoison, but have recently realised it is plagued with timeouts. It became such a problem for some that someone developed an extension to retry (as that seemed to solve most of the problems):

Issue which mentions changing version of Hackney helped:

Has anybody else experienced these issues with HTTPoison? I will be making around 300 concurrent requests/second (to start), and so want to reduce failure build-up as much as possible.

Tesla appears to have retry middleware built into it, and the ability to switch the adaptor (httpc, ibrowse, hackney).

I know the only real answer is to test each of these libraries under load, but I’d be interested to hear if anyone has run into any of the issues mentioned above. I can’t seem to find any information on why these timeouts sporadically happen. Don’t know if it’s the library or the adaptor or some lower-level processing.

As an aside - are there other options to increase the reliability? Perhaps using the adaptors directly? Or offloading the work to a more battle-tested library in another language?

2 Likes
2

There is lpgauth/buoy if you really need high performance (comes at a cost).

Or offloading the work to a more battle-tested library in another language?

I’ve only heard about puzza007/katipo which uses libcurl.

1 Like
3

Good to know! Thanks. I’ll look into those. Might be able to accept those costs.

4

If you are making requests to servers running nginx (or any other sane web server), then you should be okay, I think. I’ve used buoy to interface with google cloud storage api and it worked well.

I’ve never used katipo.

5

The requests could be going to any server, it’s not an internal tool. Hmm… cheers, I’ll think it over a little more. Looking at the benchmarks, there was a 0% failure rate in almost all cases, including with Hackney. Makes me think something else is up - they are using 256gb ram. One of the causes of the random timeouts in Hackney was something to do with how it was managing internal queues. So could be a memory issue.

I think I’ll have to load test these libs myself. I think that’s the only way I can be sure.

6

I have used HTTPoison in my production server and I am too plagued with unknown intermittent timeout issues even though I am already running the latest hackney. @BitGonzo Just wondering which library you go for in the end?

2 Likes
7

Sounds like a Hackney issue. I’ve been using Tesla and the default adaptor (httpc) and not had any problems. Have wrapped all of my requests so that I can switch out the library if it becomes a problem in the future.

Our loads will likely vary so YMMV, too.

8

I almost always start out with httpc which is built right into erlang. The api is kind of eh because it’s built for Erlang instead of Elixir, but it has never done me wrong.

9

It doesn’t verify certificates though last I checked?

1 Like
10

Correct. Most HTTP clients still inherit :ssl’s default of verify: :none, meaning connections to HTTPS URLs silently ignore the server’s certificate:

iex(1)> :httpc.request('https://untrusted-root.badssl.com/')
{:ok,
 {{'HTTP/1.1', 200, 'OK'}, ...}}

At least in recent versions, when passing in verify: :verify_peer and a trust store (e.g. from the certifi package) both chain validation and hostname verification work out-of-the-box:

iex(2)> :httpc.request(:get, {'https://untrusted-root.badssl.com/', []}, [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile]], [])
{:error,
 {:failed_connect, ...}}

iex(3)> :httpc.request(:get, {'https://wrong.host.badssl.com/', []}, [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile]], [])    
{:error,
 {:failed_connect, ...}}

(There are some known issues with cross-signed and wildcard certs, though; I would recommend using the ssl_verify_fun package for maximum server compatibility)

Edit: the above testing was done with Erlang/OTP 21 (inets 7.0)

9 Likes
11

I am suspecting its hackney issue too, but I really like HTTPoison default ssl options and connection pooling (hackney) but the timeout issue is really a dealbreaker. httpc also seemed to be slower for me as well as my use case require me to have concurrent TCP connections.

12

Two fairly new HTTP clients to choose from: Gun (for Erlang, recently hit 1.0.0) and an Elixir wrapper called MachineGun using poolboy:

Anyone had a chance to try them out?

6 Likes
13

Katipo has the advantage of simply leveraging libcurl in this regard. See the tests

badssl(_) ->
{error, _} =
    katipo:get(?POOL, <<"https://expired.badssl.com/">>),
{error, _} =
    katipo:get(?POOL, <<"https://wrong.host.badssl.com/">>),
{error, _} =
    katipo:get(?POOL, <<"https://self-signed.badssl.com/">>),
{error, _} =
    katipo:get(?POOL, <<"https://untrusted-root.badssl.com/">>).
14

Dude this is amazing, I love you. I have a project now where I need 300-400 requests per second and I’m having dififulty because they dont support http1.1. This should help a lot I think.

Katipo also looks cool as it just wraps curl

2 Likes
15

I dont’ know why i cant edit my post, but i meant http/2. obviously they support http/1.1 or id be in bad shape lmao

2 Likes