Library configuration - multi-instance

Generally, libraries shouldn’t rely on application env at all. A lot of libraries continue to use that pattern because it was popular in the early days of Elixir. But its a bad model and shouldn’t be copied. If your library needs to use processes, then users should be able to pass in arguments when they start the process in their supervisor. Redix is an excellent example of how to do this well.

There are exceptions to this rule. Sometimes it makes the most sense to provide a full-blown OTP Application. I typically do this because there’s nothing in the application to configure, the supervision hierarchy is complex, or if there’s no benefit to the user managing the lifecycle of the process.

Building a library that is re-usable without collisions is often much more difficult. But I think it pays off in the long run.

I think your suggestion focused on the “config.exs” part. Your suggestion is to do something like:

defmodule MyApp.Application do
  @moduledoc false

  use Application

  def start(_type, _args) do
    children = [
       {MyLibrary, my_library()}
    ]
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  defp my_library do
    # my library options
  end
end

This helps with: any System.get_env() (or fetch_env()) will work on any environment as this is called on application boot. Although this does not say much about multiple instances.

With that it also makes it a bit more inconvenient for per-environment settings. Think you have a client of a service that has a sandbox and a production environment. You want to hit sandbox while you are running locally, but on your deploy you want to hit production. In this scenario you would probably delegate to config.exs although using your own otp_app.

With multiple instances it would become something like:

defmodule MyClient1 do
  use MyLibraryClient, otp_app: :my_app
end

defmodule MyClient2 do
  use MyLibraryClient, otp_app: :my_app
end

defmodule MyApp.Application do
  @moduledoc false

  use Application

  def start(_type, _args) do
    children = [
       {MyClient1, my_credentials_1()},
       {MyClient2, my_credentials_2()},
    ]
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  defp my_credentials_1 do
    # my library options
  end

  defp my_credentials_2 do
    # my library options 2
  end
end

I think 99% of the libraries are not using this approach currently ad maybe we should add some link on the Elixir page about that. Also, this is maybe some overhead for a process-less library (IMHO I’d still think this is meaningless overhead).

Also, that might make using this library as a depency of another library a little harder.

When you say:

Building a library that is re-usable without collisions is often much more difficult. But I think it pays off in the long run.

You mean library authors think we should always account for the multiple instances? That is my current opinion too.

Pigeon’s config might also be interesting for you to check out.. GitHub - codedge-llc/pigeon: iOS and Android push notifications for Elixir · GitHub

eg. config from DB - API for multiple config/instances https://hexdocs.pm/pigeon/apns-apple-ios.html#custom-worker-connections etc.

I should elaborate here. If you allow the user to pass arguments through child_spec then the user is really free to start as many processes as they want. Using Redix as an example, the user is free to do this:

def start(_type, _args) do
    children = [
       {Redix, name: :primary},
       {Redix, name: :secondary},
    ]
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

Its common for people to use config.exs for this but its not necessary to do so. You can just as easily do something like this:

def start(_type, _args) do
  children = [
     {MyClient1, adapter: adapter()},
  ]

  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

def adapter do
  if System.get_env("MIX_ENV") == "prod" do
    DBAdapter
  else
    Sandbox
  end
end

In most of our apps we use vapor for reading environment config but it boils down to the same thing. If you don’t want to pass the adapter module as args you could just as easily use Application.put_env to store the correct adapter to use.

The reason that ecto needs the adapter specified in config.exs (at least historically) is because its using macros to inject the correct functions into your Repo module. Personally I don’t think the pattern that Ecto.Repo or Phoenix.Endpoint use should be emulated without a very compelling reason.

FWIW the elixir library guidelines have some points on this as well.

I think your strategy helps us avoiding using a macro. You’d start the supervision tree of the library with the name passed in (making it a mandatory option) and all calls in the library would pass that name, right?

Its common for people to use config.exs for this but its not necessary to do so. You can just as easily do something like this:

I think that using a compile environment variable for a runtime decision this is tricky and not proper in my opinion. Sandbox is an application environment and not a compilation/configuration environment and the user’s app would probably be compiled with MIX_ENV=prod to hit sandbox. So, I think that in your MIX_ENV=prod there would be calls to System.fetch_env and I think it is a valid use for config.exs.

I did not know about vapor! Thanks for poiting that out

Sorry, I only used MIX_ENV as an example. You could use any environment variable. I would also argue that choosing between Sandbox and DBAdapter is only a compile time concern if your lib is compiling the adapter into the Repo module. Otherwise the adapter you use will still be determined at runtime. For instance if you do this:

# config.exs
config :my_lib, MyRepo,
  adapter: Sandbox

defmodule MyRepo do
  defp adapter do
    Application.get_env(:my_lib, MyRepo)[:adapter]
  end
end

The adapter is still chosen at runtime. At that point there’s not a lot of difference between calling Application.get_env and passing the adapter module as an argument like: Repo.start_link(adapter: Sandbox). If you pass the adapter as an argument then you have no coupling to Application at all, which is nicer for end users.

That said, I’m not necessarily opposed to using config.exs for configuring the adapter. If you go that route then you need to make sure you don’t do something like this:

config :my_lib, adapter: Sandbox

That configuration will be global across all instances of your library and will stop your users from running multiple instances of your library. This is the mistake that most elixir libraries make. Instead, you’ll need to use a configuration like:

config :my_lib, MyInstance,
  adapter: Adapter

The user will need to understand how to wire all of that up correctly. The situation gets even more gnarly if other libraries want to use your library. In that scenario, the user will have to configure instances of your library that they’re using, and configure instances that the other library is using.

In the end, passing arguments is always more flexible than trying to use Application configs, and its why I default to just passing arguments in most cases. Using config.exs has its place but it’s much less flexible and I think it should be used sparingly. The majority of libraries that use config.exs shouldn’t.

Completely agree with passing arguments as an always more flexible approach.

Just wanted to comment that on your example it still is a compile time dependency. I quote with my comments:

# config.exs
config :my_lib, MyRepo,
  adapter: Sandbox # value is determined at compile time

defmodule MyRepo do
  defp adapter do
    # Even if next line is called during app boot, it uses a compile time value
    Application.get_env(:my_lib, MyRepo)[:adapter]
  end
end

The only way to have a true runtime (which here means a value provided where the application is run) is by using anything only available at the place where the app is run. This is why I’ve mentioned System.fetch_env but could be something like a configuration provider only available at deployment environment.

In any case, your pattern is a very clear one: the library should receive all of its configuration as parameters and, if there is any process, use the application supervision to get those values. With that we avoid the config.exs where we can. Also, for multiple instances we can use your strategy of having several processes under the supervision tree, each with its own name passed as an argument.

Just as an example (I know I’m verbose… please bear with me…):

defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
       {MyLibrary, my_credentials_1()},
       {MyLibrary, my_credentials_2()},
    ]
    # ...
  end

  defp my_credentials_1 do
     [
       name: :my_library_credentials_1,
       # other options
     ]
  end

    defp my_credentials_2 do
     [
       name: :my_library_credentials_2,
       # other options
     ]
  end
end

With that we have:

• MyApp.Aplication → MyApp.Supervisor
• MyApp.Superversion → :my_library_credentials_1
• MyApp.Superversion → :my_library_credentials_2

This seems like the best approach I guess. The library would use calls with names like MyLibrary.some_function(:my_library_credentials_1) and fetch options with that.

I just wonder now if there should be some default name to avoid needing an extra argument when there is only one configuration available. The same example call would then be MyLibrary.some_function() and that would use the :default configuration name as a default argument.

Thanks once again for the discussion!

Let me see if I can clarify this a bit. If we have this code:

config :my_lib, MyRepo,
  adapter: Sandbox # value is determined at compile time

defmodule MyRepo do
  defp adapter do
    # Even if next line is called during app boot, it uses a compile time value
    Application.get_env(:my_lib, MyRepo)[:adapter]
  end
end

The adapter isn’t actually being “compiled” into the MyRepo module. Its always being fetched from the application env. That means its possible to do something like this:

# The adapter is set from when `config.exs` was initially run
Sandbox = MyRepo.adapter()

# Update the config...
Application.put_env(:my_lib, MyRepo, [adapter: DBAdapter])

# Now the adapter has changed
DBAdapter = MyRepo.adapter()

Its possible to compile the adapter directly into the module doing something like this:

defmodule Repo do
  defmacro __using__(_opts) do
    quote do
      @before_compile unquote(__MODULE__)
    end
  end

  defmacro __before_compile__(env) do
    a = Application.get_env(:my_lib, Repo)[:adapter]

    quote do
      def adapter do
        unquote(a)
      end
    end
  end
end

defmodule MyRepo do
  use Repo
end

This will cause the adapter function to be hard-coded with the adapter module into MyRepo.

Hopefully that provides some clarity :).

Everything else seems spot on to me .

In my applications and services I’ll often use this pattern:

defmodule Server do
  def start_link(opts) do
    GenServer.start_link(__MODULE__, state, name: opts[:name] || __MODULE__)
  end

  def foo(server \\ __MODULE__, args) do
    GenServer.cast(server, {:foo, args})
  end
end

This gives me a lot of flexibility when testing the server and provides reuse. But, I don’t tend to use this pattern in libraries because it can create name conflicts if 2 different users try to use the default name. Passing the name around can definitely be annoying, but I haven’t found a better pattern that still provides the same level of flexibility. At this point I’m used to passing the name or a pid as the first argument to my function calls.

Likewise