Mix config evolutions

init runtime configuration only works for supervised dependencies, it will not work for things like:

  plug Plug.Session,
    signing_salt: "how to manage this setting"

Maybe applications could accept a module instead of values (or both) ?

  plug Plug.Session,
    handler: MyApp.MyConfigHandler

defmodule MyApp.MyConfigHandler do
  # use MyApp.CompileTimeSettings
  def setting({:plug, :session, :cookie, :signing_salt}) do
    # - "very secret signing salt"
    # - System.get_env("COOKIE_SIGNING_SALT")
    # - MyExternalServiceAPI.get(:signing_salt)
    # - or whatever
  end

  def setting(tuple) do
    IO.inspect tuple, label: "Show me available settings"
    nil
  end
end

I would be very happy to be able to configure my dependencies this way.

Hi, I have a few questions on the direction that’s been mentioned above.

The two problems I’ve found so far with {:system, "MY_VAR"} are that:

• it’s not standard and requires custom code (that people are extracting in libraries)
• Reading with System.get_env("MY_VAR") is slower than reading from the Application env (but benchmarks show that caching it in the wrapper speeds thing up).

Both problems could be solved by adding a layer between the Elixir API and the underlying Erlang calls.

So I guess that a different way to frame the problem is that we have a centralized configuration API that does not currently support runtime configuration, which is a common requirement, as opposed to saying that “runtime configuration should be moved to runtime”.

I can see how this would work when starting supervisors and processes. If a dependency requires configuration (e.g. API tokens, URLs, flags, etc), they can be read from the ENV and passed to the supervision tree, then something will either store the config in a GenServer’s state or a ETS table, depending on how frequently it needs to be accessed. (please correct me if I’m wrong)
I am not sure how it would work for libraries that don’t start processes though.

The changes happen on two different levels.

Internally: it means that Ecto will internally expect configurations to be set at runtime rather than compile time.

Externally: it means that Ecto will provide a proper API to do runtime configuration. The only way to do runtime configuration in Ecto 2.0 was by using url: {:system, "DATABASE_URL"} which has two big problems: it only works for the :url parameter and it only allows the configuration to be read from the system environment. If you need to read it from elsewhere, you are out of luck.

Ecto 2.1 now provides the init/2 callback inside your repo for runtime configuration:

def init(_, config) do
  {:ok, Keyword.put(config, :url, System.get_env("DATABASE_URL"))}
end

Now you have the flexibility to do whatever you want. Also note Ecto allows the configuration to be given at the moment you call start_link on the Repo.

Can you explain a bit more? As far as I can tell, Ecto is configured using Mix.Config and a config.exs file.

From the Ecto documentation (2.1.4):

Where the configuration for the Repo must be in your application environment, usually defined in your config/config.exs

It says “usually”, but are there other options? If so, the documentation doesn’t mention them.

How can I configure my repo without using a config.exs file?

Thanks.

I think if we have a constructive discussion we should be able to steer in a direction. This particular issue seems important enough for an “official” statement. The library ecosystem is what makes a language great. If we can have some standard it will help the language and its users.

Yes, exactly. Rely less on compile-time configs and more on arguments. The application environment is global configuration and therefore it can be prone to conflicts.

Maybe I was not clear on my first reply but that’s exactly the direction Ecto and Phoenix are moving. We will still use the application environment by default for convenience but we are moving as much as we can to runtime and consequently also allow those to be given as options.

Wasn’t expecting this. However, having libraries that configure themselves through an application environment is a given. I wonder if we, as a community, have enough weight to add another given (configure through functions and options - which in itself is an excellent idea). Or should we make the given a nice, first-class citizen? (at least nicer than it is now, where probably the biggest single issue is application startup order where your “library” applications are started before you have a chance to setup their environments)

Having libraries that are inflexible in their configuration because they have an opinion on what the config.exs looks like can indeed really be a deal killer. I ran into this recently in an Elixir project and it was one of those “oh, no..” moments . .. and as people have noted already, different projects take different approaches, meaning the ability to learn A Way™ and repeatedly apply that to Elixir projects is not really so possible right now. Having a clearly stated, documented, and followed by the “core”/“popular” libraries/tools would be absolutely fantastic. 1000 upvotes for that suggestion.

That said, it would really be nice imho to have a standardized schema mechanic as OP suggested. Libraries may not necessarily use that schema directly, but it could be used by application developers to easily map to/from storage solutions (be they POT files in the file system, records in a key/value store, rows in a tennantized SQL database, ..) and give some sense as to what the options are (allowed values, etc) in a way that could be turned into automated tests at runtime.

It would also allow some level of decoupling between the actual MFA calls to configure a library and the configuration, which is ultimately data, not an API, and the configuration management.

With some sort of schema mechanism in place, then application-side (or node orchestration level) tools could evolve independently of library devel (targetting different use cases / environments, even) using the schemas as their generic integration point rather than “random” MFAs in libraries. This would be far more realistic a target imho than semi-de-facto-standardized-as-best-practice-ideology. It would even provide a(n optional) path for library developers to have “best practice” configuration functions generated at compile-time for them.

If these schemas were similar to ecto schemas, allowing for knowledge transfer / consistency / even some potential interoperability, that would be icing on the cake to me …

tl;dr from the perspective of an application developer (and knowing the pain points our ops team faces), I would love to see a configuration schema DSL for library developers to provide for application developers and deployment management.

The goal of this thread is not to throw anything out of deprecate config.exs, but to list issues and find solutions.

The first thing that seems to come out is that we should encourage lib author to take configuration via arguments when possible. If this is accepted by @josevalim, a bit of documentation and “word spreading” would be enough for this one.

On the other hand, the app configuration (conform, mix config…) needs to be addressed in some way.

To be clear, I wasn’t suggesting avoiding libraries just because they’re taking options through app env At my work, we certainly use many such libs, and then do some trickery to vary the settings at runtime.

However, my understanding is that this thread is an attempt to summarize config issues and discuss potential solutions:

Given that direction of the discussion, I believe that many (though admittedly not all) issues would be resolved if libraries accepted options as parameters. So I think that rather than trying to invent some complex schemes in e.g. mix, a better approach would be to educate library authors to avoid requiring app env where it’s not needed. Therefore, I agree with this conclusion: