BitGonzo

BitGonzo

Reliable HTTP libraries? (HTTPoison, HTTPotion, Tesla)?

There was a post made quite a while back regarding the first two, which made mention of Tesla:

I originally started using HTTPoison, but have recently realised it is plagued with timeouts. It became such a problem for some that someone developed an extension to retry (as that seemed to solve most of the problems):

Issue which mentions changing version of Hackney helped:

https://github.com/edgurgel/httpoison/issues/291

Has anybody else experienced these issues with HTTPoison? I will be making around 300 concurrent requests/second (to start), and so want to reduce failure build-up as much as possible.

Tesla appears to have retry middleware built into it, and the ability to switch the adaptor (httpc, ibrowse, hackney).

https://github.com/teamon/tesla#direct-usage

I know the only real answer is to test each of these libraries under load, but I’d be interested to hear if anyone has run into any of the issues mentioned above. I can’t seem to find any information on why these timeouts sporadically happen. Don’t know if it’s the library or the adaptor or some lower-level processing.

As an aside - are there other options to increase the reliability? Perhaps using the adaptors directly? Or offloading the work to a more battle-tested library in another language?

Most Liked

voltone

voltone

Correct. Most HTTP clients still inherit :ssl’s default of verify: :none, meaning connections to HTTPS URLs silently ignore the server’s certificate:

iex(1)> :httpc.request('https://untrusted-root.badssl.com/')
{:ok,
 {{'HTTP/1.1', 200, 'OK'}, ...}}

At least in recent versions, when passing in verify: :verify_peer and a trust store (e.g. from the certifi package) both chain validation and hostname verification work out-of-the-box:

iex(2)> :httpc.request(:get, {'https://untrusted-root.badssl.com/', []}, [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile]], [])
{:error,
 {:failed_connect, ...}}

iex(3)> :httpc.request(:get, {'https://wrong.host.badssl.com/', []}, [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile]], [])    
{:error,
 {:failed_connect, ...}}

(There are some known issues with cross-signed and wildcard certs, though; I would recommend using the ssl_verify_fun package for maximum server compatibility)

Edit: the above testing was done with Erlang/OTP 21 (inets 7.0)

svilen

svilen

Author of Concurrent Data Processing in Elixir

Two fairly new HTTP clients to choose from: Gun (for Erlang, recently hit 1.0.0) and an Elixir wrapper called MachineGun using poolboy:

https://github.com/ninenines/gun
https://github.com/petrohi/machine_gun

Anyone had a chance to try them out?

steven7

steven7

I have used HTTPoison in my production server and I am too plagued with unknown intermittent timeout issues even though I am already running the latest hackney. @BitGonzo Just wondering which library you go for in the end?

Last Post!

9mm

9mm

I dont’ know why i cant edit my post, but i meant http/2. obviously they support http/1.1 or id be in bad shape lmao

Where Next?

Popular in Discussions Top

restack_oslo
Hello, Please pardon me for any faux paux. I am 46 and this is my first time on a forum of any kind. I wanted to to get answers from tho...
New
pillaiindu
In django there is a cache framework backed by memcached. Rails also puts a lot of emphasis on caching, and even the idea of russian-doll...
New
acrolink
How does the two languages compare when it comes to server side application development? Any experiences or ideas? Thank you.
New
New
chulkilee
Here are the list of HTTP client libraries/wrappers, and some thoughts on HTTP client in general. I’d like to hear from others how they w...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36654 110
New
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42533 114
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement