HTTP client libraries - need better one in standard library?

This is something that was on my mind for some time now. We have the situation repeating that I saw in Ruby ecosystem as well, where it hasn’t been exactly solved: I end up with projects having dependencies on multiple HTTP client libraries.

The reason behind people wanting to use third party library in first place, is that it requires significant legwork to get it done right using standard Erlang standard library: if you want ssl, gzip, redirects etc. It also does not feel exactly Elixir idiomatic to use it.

So developers pick up whatever library they feel is slightly better/faster to use out of the box. I understand that making HTTP requests only looks like simple task, and in fact is a complex scenario that varies from use case to use case, but having a decent, easy to use HTTP client in standard library would bring in many benefits:

  • we can reduce dependencies on our projects
  • we have central place to control/monitor/rate limit/stub HTTP requests we’re making
  • we have a number of expected errors/exceptions that we can expect to happen, esp. when third party API wrappers do focus on happy paths only

What are your thoughts?

8 Likes

I don’t see any intrinsic value in making it part of the standard library. The only thing that would be reduced is the amount of explicit dependencies, but using it would still incur the same cost anyway.

What is it you imagine will be better about point 2 and 3 with regards to a standard lib offering? I don’t see how it’ll be any different than pulling in a good dependency (HTTPoison, for example) and using that. There’s nothing privileged about modules that are shipped with the standard distribution.

2 Likes

Including in stdlib, has a huge consequence that is generally ignored - it has to be maintained from that point onwards by the core team. There’s only so much time in a day, so ultimately this means less maintenance for other parts of stdlib.

The bigger problem is that there’s no clear winner in terms of an http library with a good interface (by good interface, I mean a composable one).

8 Likes

My point was that from a technical standpoint no module that is included in the standard distribution is privileged; it’s a module like any other. As such, it won’t matter that it’s in the standard distro.

1 Like

There is a HTTP client in the standard library !

It comes with the BEAM and it is not amazing at all.

To be honest, it even shows the problems of stdlib for that :

  • the whole thing is old
  • it can only evolve through OTP official releases, and they are not going to do a release just for that
  • It is not “central” to what OTP is, so it is always a “side project” so it does not get that much support from the team, they did not enlist for that
  • any update to it need to super solid because a problem means updating the whole BEAM for everyone
  • etc etc

There is hope on this front mind you : with the coming separation of the OTP repo into separated app, we may end up getting a bit of a revival for PR on it. But i would not bet too much on it.

Another example of that types of problems is the webserver inside erlang (httpd) the xml parser (xmlerl) etc etc

3 Likes

This^

There are several Erlang HTTP clients for this reason, the standard client is oldish and the other ones are plain better. There are some which are also built on top of NIFs for better performance.

1 Like

The main problem I see with bringing one of existing Elixir HTTP clients as one into standard library is actually related to that. The clients I saw rely not on standard httpc client, but use third party libraries. So we have third party Elixir library, pulling third party Erlang library, to do HTTP requests.

1 Like

Yes and exactly because putting it in the standard library do not work

5 Likes

HTTPosion is a total disaster. After 25,000,000,000 requests in last few months and looking up for internal structure of hackney and the lib, I concluded that this lib stops working (even simple get requests). I replaced whole code with Tesla and everything worked.

I couldn’t find single piece of useful error code but “connection error”.

3 Likes

seems like an excellent find :slight_smile: thank you.

7 Likes

Tesla with which adapter?

1 Like

With Native Erlang one

1 Like

That sounds like this Hackney issue that is being worked on.

https://github.com/benoitc/hackney/issues/420

1 Like

And what’s the problem with oldish lib? it’s stable. The whole point of using Erlang is to be in a stable state. :077:

2 Likes

Hi, Is there a HTTP client that is written only in Elixir. just relying on components from the standard library. i.e. not hackney or ibrowse

Would you count a wrapper around :httpc as pure elixir?

http://erlang.org/doc/man/httpc.html

Anyway, since we do not have wrappers for erlangs TCP modules, pure elixir implementations are not possible unless you manage to implement a pure HTTP library and do not care for transportation…

I would as httpc is part not an extra dependency.

If you don’t like the one included in Elixir, it’d be a lot of work for the core team to maintain even more which time constraints limit the ability to do, why not use a third-party one or write your own?

The one that ships with Elixir is included in erlang, isn’t maintained by the Elixir core team and there is no HTTP client that is.

If you’re responding to @Crowdhailer: What @Crowdhailer is asking about is if there is one implemented only in Elixir without any non-BEAM dependencies.

Exactly.

Which is what I am doing :-). Am currently working on the HTTP/1 side of the client in Ace

1 Like