Proper way of making a module for an optional (default) HTTP adapter

Questions / Help
1

I am writing a library that is essentially an HTTP API client, and I’m using Req as my default HTTP adapter. However, I don’t want to enforce Req as the only possible adapter, so I want to make the library’s user to be able to use their own custom HTTP adapter. Hence, I make Req an optional dependency, so it isn’t downloaded and compiled if it’s not used.

The module that implements the behaviour and essentially handles the use of adapter looks like this:

defmodule MyApp.HTTPClient do
  @http_client Application.compile_env(:my_app, :http_client, MyApp.HTTPClient.Req)

  @callback request(term()) :: term() # subject to change, obviously, as I finalize the adapter
  def request(arg), do: @http_client.request(arg)
end

But I am having some trouble when it comes to implementing the default Req adapter and also polishing the overall custom adapter logic.

If the app is configured to use Req, but Req isn’t available, I feel like I need to raise an error at compile time, as opposed to during runtime. And also I think I should not compile the module and clutter the namespace if Req isn’t used. So I came up with something like this:

if !Application.compile_env(:my_app, :http_client) do
  Code.ensure_compiled!(Req)
  defmodule MyApp.HTTPClient.Req do
    @behaviour MyApp.HTTPClient
    # module logic here
  end
end

However, I’m not sure if this is good practice to have such checks outside of module definitions. Besides, I’m afraid (maybe unreasonably) that this might break compilation order, optimizations and maybe something else. And, to be honest, it simply looks ugly and unconventionally.

I have also considered the following solution:

defmodule MyApp.HTTPClient.Req do
  @behaviour MyApp.HTTPClient
  @req Code.ensure_compiled?(Req)
  
  if @req do
    @impl true
    def request(arg), do: Req.new ...
  else
    @impl true
    def request(_), do: raise "Req is not available..."
  end
end

But this obviously only works during runtime and possibly warns the user when it’s already too late. The error may even go unnoticed. And when my behaviour grows, I will have to implement that error for every single function, which is not optimal as well. And this might break callback compliance too.

Also, if the user defines their own adapter, I want to check whether it’s available and if it’s behaviour-compliant. And, if possible, if Req isn’t installed, but a custom adapter is not defined either, automatically installing Req would be nice.

So, what would be the proper way to accomplish all this? I am somewhat experienced in Elixir, but I’ve never had to deal with such problems before. I’ve taken a look at Tesla source code, but it’s different: Tesla checks if an adapter is installed, rather than configured to be used. Also, I want to make it clear that I am going to write proper and explicit documentation, but we all know how this usually goes, so I still want all those compile-time checks.

2 Likes
2

I tend to use Code.ensure_loaded?(Req) as a condition to see if a dependency is available or not. That’s also what I noticed many libraries do. With conditional compilation of the adapter you can then have a second place make sure the configured adapter is actually available.

3

Why? Seems like a way to make your life much harder.

4

Why don’t you use a protocol? Behaviours are used for filling in holes in process machinery. You’re looking for an interface to program against. With a protocol you can pack additional data in the implementor and do all kinds of interesting things.

5

Unfortunately, I am having a hard time trying to come up with a solution that uses protocols. So far I’ve only implemented protocols for structs.

6

So, do you suggest something like this?

if Application.compile_env(:my_app, :http_client) = MyApp.HTTPClient.Req and Code.ensure_loaded!(Req) do
  defmodule MyApp.HTTPClient.Req do
  # ...
  end
end

Elixir appears to be rather powerful when it comes to such things, but I’m constantly constraining myself because I feel like something is not really traditional or is the proper way to implement stuff.

7

I don’t think it makes my life any harder. All I need to do is to implement the option to provide a custom client, then it’s up to the user to actually make that client work. Why? Because, for example, an user might want/need to use Hackney, which might have some features that Req doesn’t have.

Now let’s be real, I’m pretty sure quite literally no one is going to use my library but myself, as I’m writing it for a rather niche service, but I still want to make it flexible and "proper”.

8

IMO you are trying to optimise for a use-case that might very well never occur. In your given example, if you discover Req does not have an important feature while some other HTTP client does then most of us would do our best to either contribute to Req or just point out the deficiency to the authors.

Your approach would encourage segmentation. In the library I am about to release I am fighting tooth and nail to have every feature I wanted to work reliably – and this has costed me multiple weekends back in spring 2025.

At the end of the day it’s your thing and you’ll manage it how you deem best – obviously. But to me that’s one of those extensibility traps that I have fallen into before and do my best to never do so again. It’s super rarely worth it.

Finally, a compromising solution would be to introduce a plethora of configuration options in your library that you could re-translate and pass to Req. I have found myself doing that once and while it was clunky and annoying (and consumed much more time than I wanted), it still got the job done well and helped a future contributor use a different HTTP client down the line.

9

Req already comes with the option to replace the underlying http client adapter. It just defaults to finch. So if you allow users to customize that setting you can continue to use Req directly.

2 Likes
10

Yes but it actually doesn’t support other adapters at the moment.

Valid point, for example the user might need to connect through a HTTP or socks proxy and this might not be supported by Mint. I remember having to switch to :httpc for this reason.

But then why not using Tesla? It supports all possible adapters, the adapter can be configured globally by the library user and problem solved :slight_smile:

I don’t intend to launch a debate about Tesla vs Req, but:

  • req is pre 1.0
  • any sizable project have both req and tesla as dependencies anyway, along with mint, finch, HTTPPoison and hackney. Yes this is a bit messy
1 Like
11

Yet another HTTP client ?

I would find and use the one you like directly, there is also a project which already does what you did. It is called Tesla.

Anyways, at this point, I don’t think using Tesla is a good idea because Req is becoming standard the facto, being used in Phoenix as a standard Http client and so many other projects.

1 Like