Proposal: moving towards discoverable config files

releases
elixir

#1

One of the major differences between running your application as a release and as a Mix project is the differences in configuration. Mix evaluates the configuration right before the application starts, releases evaluates the configuration when your application is compiled.

This implies in a large mismatch of how those two environments are used. For releases, environment variables (read by System.get_env/1) need to be set when the application is compiled and such information may not be available at this point.

Ideally, we would want a release to evaluate the configurations files in config when the release starts. One approach would be to copy the configuration files as is to the release but that’s hard to achieve in practice for two reasons:

  1. A config file may import other config files and often importing those files happen dynamically. For example: import_config "#{Mix.env()}.exs". The dynamic import makes it hard for release tools to know which configuration files must be copied to a release, especially in cases like umbrella projects, where a developer may load configuration across projects

  2. Even we copy today’s configuration files to a release, those configuration files rely on Mix, which is a build tool and therefore it is not available during releases

To solve those issues, we need to make sure we can discover all imports of a configuration file without evaluating its contents. We also need to introduce a new module for configuration that does not depend on Mix.

This is the goal of this proposal.

Application.Config

This proposal is about introducing a module named Application.Config. It will work similarly to the existing Mix.Config, except it belongs to the :elixir application instead of :mix. This allows releases to leverage configuration without depending on Mix.

The user API of Application.Config is quite similar to Mix.Config. There is config/2 and config/3 to define configurations. There still is import_config/1 to import new configuration files with one important difference: the argument to import_config/1 must be a literal string. So interpolation, variables or any other dynamic pattern is no longer allowed.

In order to help with configuration management, we will introduce a project option in your mix.exs, named :config_paths to help manage multiple required and optional configuration files.

In the next section we will provide an example of how configuration files used by projects like Nerves and Phoenix will have to be rewritten and then we will discuss how integration with release tools such as distillery will work.

A common example

Projects like Nerves and Phoenix generate files with built-in multi-environment configuration. Today, this configuration has an entry point config/config.exs file that imports an environment specific configuration at the bottom:

# config/config.exs
use Mix.Config

config :my_app, :some_shared_configuration, ...

import_config "#{Mix.env()}.exs"

And then each config/{dev,test,prod}.exs provides environment specific configuration. For instance:

# config/dev.exs
use Mix.Config

config :my_app, :some_dev_configuration, ...

The issue in the example above is the use of dynamic imports, such as import_config "#{Mix.env()}.exs". We will address this by defining both config/config.exs and config/#{Mix.env()}.exs as configuration entry points in your mix.exs:

# mix.exs
def project do
  [
    ...,
    config_paths: ~w(config/config.exs config/#{Mix.env()}.exs),
    ...
  ]
end

And now we can define those configuration files without dynamic imports:

# config/config.exs
import Application.Config

config :my_app, :some_shared_configuration, ...
# config/dev.exs
import Application.Config

config :my_app, :some_dev_configuration, ...

In Phoenix, the config/prod.exs case may link to a separate prod.secret.exs file. While we could also refer to this file in the :config_paths configuration in the mix.exs file, because it is only specific to production, it is more straight-forward to continue importing it at the bottom. So a config/prod.exs would look like this:

# config/prod.exs
import Application.Config

config :my_app, :some_prod_configuration, ...

import_config "prod.secret.exs"

By adding :config_paths, we are able to move the dynamic configuration to the mix.exs file and make the order that configuration files are loaded clearer.

A FarmBot example

Nerves projects tend to rely extensively on configuration files. So let’s look into existing open source Nerves projects and see how this proposal will fare. Let’s take a look at FarmBot v6.4.1.

The questions we want to answer are: if we move the FarmBot project to the proposed Application.Config, will they be able to express of all the existing idioms they do today? And, even further, will their configuration files become simpler or more complex?

From looking at its config/config.exs, we can already see a pattern that won’t work in releases: the use of Mix.env and Mix.Project.config.

We can see those variables are used to dynamically import configuration, which Application.Config won’t allow.

Those idioms are perfectly fine with how configurations work in Mix today. But they will no longer with a release built on top of Application.Config.

The solution is to move all of those imports to the :config_paths option in mix.exs. However, note that some of those dynamic imports are optional, so we will also need the ability to explicitly tag them as such:

# farmbot/mix.exs
def project do
  [
    ...,
    config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++
                   optional_config_paths(@target, Mix.env()) 
    ...
  ]
end

defp optional_config_paths("host", env),
  do: [{:optional, "config/host/#{env}.exs"}]

defp optional_config_paths(target, env),
  do: [{:optional, "config/target/#{env}.exs"}, {:optional, "config/target/#{target}.exs"}]

We believe this approach is an improvement to the previous one because it allows all environment and target specific handling to remain in the mix.exs file and not scattered around multiple configuration files.

Using it in releases

In the previous sections, we have outlined Application.Config which no longer depends on Mix and has a restricted import_config.

Now that we are able to see all of the configuration files that affect our system, a release tool, such as distillery, should be able to traverse all of those configuration files and merge them into a final config/release.exs that will be part of your release. In fact, Elixir will provide a convenient API that performs such operation, streamlining the release assembling process.

Unresolved topics

There are two important topics that we have not included in this proposal and they will be discussed in a further step.

  1. What about umbrella projects? Umbrella projects also rely on configuration and we need to make sure the listed mechanisms also work well with umbrellas.

  2. How to avoid common pitfalls? Even though we will migrate to Application.Config, there is nothing stopping a developer from accessing Mix (and the module defined in the mix.exs file) from their new config files. As we have seen, this may lead to errors when running releases, as releases do not have Mix available. To address this, we may introduce checks when assembling releases that make sure Mix is not invoked in configuration files, raising appropriate error messages in case they do.

Summing up

We propose a new Application.Config module and a new :config_paths project option that allows release tools to discover all of the relevant configurations in a system. Release tools can then merge and copy those configuration into releases and execute them as part of the release process, allowing dynamic calls such as System.get_env/1 to work in development and in production transparently, with or without releases.


Should we adopt Dave's way of building applications as a series of components? (Dave's talk has now been added!)
Do we need an equivalent to the Rails Credentials file?
How to use different email config for staging and production
Elixir Blog Posts
#3

Would it also make sense at this point to separate build time configuration from runtime configuration?

The example of build time configuration that comes to mind is:

Once you’ve built with a particular configuration, it no longer matters what value you’ve set it to. It also acts as a bit of a gotcha about how to pick up a change if you’ve already built once.


#4

With this proposal, would Mix.Config still exist for compile time configuration, and in what file would it be in?

What does the :optional tag mean? What does it actually do? Sorry if it’s a silly question but I didn’t really get it from the explanation.


#5

I suspect it means something like: process this file if it exists.


#6

Application configuration is a general key-value store, which means that if you do a typo on a configuration key, you may spend some time chasing why a configuration is not working only to realize it was a simple typo. If we add a distinction between runtime and compile-time, we are adding a new “vector” for misplacing configuration and a new source of confusion. Basically, we would be pushing this concern to users without giving them any support when something goes wrong.

Before we split configuration between runtime and compile time, we need to have a more declarative approach to configuration so they are easier to document and easier to check. So it is definitely something we intend to do, but we are not adding this distinction now.

It will continue to work the same. We are not making a distinction between compile time and runtime. You just put them in the same files. Just the top of the file changed from use Mix.Config to import Application.Config.

Bingo!


#7

Then how would we do actual compile time configs? For example I use this in my config.exs to set some values when building the release:

# Store app version and commit hash at compile time
config :code_stats,
  commit_hash: System.cmd("git", ["rev-parse", "--verify", "--short", "HEAD"]) |> elem(0),
  version: Mix.Project.config()[:version]

Anyway I appreciate the work on the config system, definitely an area that can use some improvement and clarification. :slight_smile:


#8

It will work the same as today: the config file will be read before your code is compiled. It will also be read before your code runs (via mix run) and also before your release starts.

You probably don’t want to run those bits inside a release though. You can control those things with the new config_paths though.


#9

Would if_exists: "path" be better than optional: "path" in that case?


#10

I see what you mean, but I think that optional for most will be simpler to remember especially where people are familiar with optional/required naming.


#11

It strikes me as potentially confusing that the same config might produce different results at compile time and runtime. It seems such a change would benefit from making the execution environment for a config expression explicit and restricted.


#12

Very interesting! I remember you guys discussing the configuration challenge during ElixirConf.EU; I am really happy that you were able to come up with such a clean looking solution! I definitely think that inversion-of-control is the way to go with configuration files. :+1:


I had the same question, especially since it was combined with the example where we have multiple function clauses based on the current building target.

I personally think that if_exists is a lot more explicit in what it checks for than optional.


Question: What would, using the new Application.Config, be the way to read out system environment variables during compilation time? Is the idea to ‘just fall back on the current Mix.Config in that case’ or will that one be deprecated at some point?


#13

Note this is already the case today. We want to improve this distinction in the future. This proposal is the first step towards this path: it allows us to better traverse configs and remove the dependency on Mix.

None of the usages and patterns will change. We are not ultimately changing how we write configs at this point, we are just allowing them to be a bit more discoverable.


#14

While I think there’s a lot of good points addressed with the above changes, I do think that keeping single system of configuration files and using it at build and run time will be confusing.

At the moment you have to do some gymnastics to get it working with replace_os_vars and special escaping of strings to support runtime configuration options. It’s not ideal.

I think that inversing the way config files are required would also be beneficial, and precisely for the case of umbrella apps where at the moment you can end up with different config when you start something from app directory than you would get from umbrella root. For the simple umbrellas many people would be happy with just top-level config files, and we could get rid of in-app-directory config files altogether. Single config to rule them all, that’d be good, while not taking away the flexibility to do it your way.

Having said that all above, when we inverse things and have the configuration evaluated by default, there will be a lot of compile-time settings being evaluated at runtime. This is not great because the user cannot immediatelly tell what is what. Let’s have a look at few examples from my config files:

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id, :application, :module, :ip, :endpoint, :uid, :oid, :type, :custom]

is this a run-time or compile-time setting?

How about Phoenix endpoint like this:

config :ui, UI.Endpoint,
  url: [host: "localhost"],
  render_errors: [view: UI.ErrorView, accepts: ~w(html json)],
  pubsub: [name: UI.PubSub, adapter: Phoenix.PubSub.PG2]

Is this compile-time, or run-time? Can you swap “host” at run-time? Can you change others too? See, this is confusing if this is going to be executed at run time and compile time alike.

At least currently you can, looking at the config files identify which are run time options (because you used escaping/env vars) on these. It’s not ideal but you can figure it out having already existing config file in front of you. Think about that new dev-ops engineer who is taking over maintaining of the project. They’ll know what the settings are by looking at the current config files - something they no longer can do if we implement evaluation of config files at run-time by default.

What I would propose instead

1. Let’s keep the current system but make it slightly more flexible, so you can actually inverse the control yourself.

This means we add config_path as proposed by @josevalim, and it would default to config/config.exs. From there, it can either include other config files (as does now) or you can walk away and inverse the control by not including other configs in that file and manipulating config_path instead. New apps, generators etc. can do it by default at some point.

2. Introduce runtime config

That can be runtime_config_paths and config/runtime_config.exs by default. That can work precisely the same as current Mix.Config, but without relying on Mix to exist, like in the proposal above. To make things consistent, I would even allow these config files to include other config files. In short: Mix.Config would just be kept as name for backward compatability, but the code itself can be moved to Application.Config so we do not maintain 2 code bases.

The runtime config is then clear place where you put your configuration that is going to be evaluated at run time, so you can expect System.get_env and similar to work here.

In dev, when you run iex -S mix it would:

  1. evaluate config/config.exs (i.e. compile-time config)
  2. compile the app
  3. evaluate config/runtime_config.exs (i.e. run-time config)
  4. run the app

while in prod, when started a release executable, where there is no mix only steps 3 & 4 would be performed.

Objective

The proposal above is 100% backward compatible. We can give time for library authors to separate their run-time and compile-time configuration options. Phoenix would generate some compile-time config in config/config.exs, but would move run time configuration to config/runtime_config.exs by default. If you still need to use some old hex package and are forced to use old-style runtime config, you will still be able to do it with this hybrid approach. But with time we get to end up with nice separation of run-time and build-time configuration options, which will make dev-ops engineer’s lifes way easier.


#15

I’m generally in favour of the proposed changes because I think they will simplify some frequent scenarios, most notably fetching settings from OS_ENV.

Another benefit is that config is now present on the server in Elixir syntax, not the Erlang one. Though, in my limited experience I can’t recall a single time when I edited sys.config in prod.

I’ve recently posted a rant on app config which was partly motivated by some discussions I’ve had with @josevalim and @bitwalker, partly by some discussions I’ve had with colleagues, and partly by my continuous distaste for ad-hoc config files (which dates back from my rails days).

It’s a long article, but TL;DR is that while I agree that config scripts are very convenient, and even in the current form flexible, the outcome is that app env tends to be bloated with all sorts of things which are not configuration at all. Even in this thread there have been a couple of examples:

config :code_stats,
  commit_hash: System.cmd("git", ["rev-parse", "--verify", "--short", "HEAD"]) |> elem(0),
  version: Mix.Project.config()[:version]

These are not system configuration parameters, they’re constants derived from the current state of code on the build machine.

config :ui, UI.Endpoint,
  url: [host: "localhost"],
  render_errors: [view: UI.ErrorView, accepts: ~w(html json)],
  pubsub: [name: UI.PubSub, adapter: Phoenix.PubSub.PG2]

As I mentioned in the article, :render_errors and :pubsub are parameters to the library, not configuration.

It could be argued that host is a system configuration, and having it here allows you to change it without needing to redeploy the system. In practice, I don’t think this is a common scenario, and even if I needed to do it, I probably wouldn’t mind redeploying the system. In other words, if you expect that the host will have to be changed and deploying will not be an option, that’s a compelling reason to make host an explicit system property. Otherwise, it’s a case of YAGNI :slight_smile:

Therefore, Instead of further expanding the support for config scripts, say by explicitly separating compile time and runtime scripts, I think that as a community we should first consider how much stuff do we place inside config for mere convenience, and as a consequence how much our config scripts and app envs become bloated with data which is not a part of the system configuration. Perhaps a better way is to promote runtime configuration from the application code (not a config script) as a recommended practice. This can’t handle all scenarios, but I believe that it can handle most of them, and that it’s a better approach.

All that said, the proposed changes look good to me, as they seem fairly limited, but as a benefit they should make hacks such as replace_os_vars obsolete, and they should in general bring OTP releases and local development closer.


#16

The proposal does not change this aspect of configuration and this exact issue exists today. To be clear, I am not saying we shouldn’t discuss it. I want to rather understand the root cause. Are we thinking this proposal makes it worse? Were folks not aware this issue existed? Or do we want to fix this issue now that we are discussing configuration?

Regarding runtime_config.exs, I dislike adding more configuration files because it is just matter of time until it becomes runtime.dev.exs, runtime.prod.secret.exs, etc. However, before we settled on this current proposal, we also discussed introducing a on_boot/1 block to configurations:

use Mix.Config

on_boot do
  config :ui, UI.Endpoint, url: [host: System.get_env("HOST")]
end

...

In this case, only what is inside the on_boot block is evaluated later. The on_boot block would also be restricted to not allow imports and similar. Both mix and releases would only run the on_boot blocks when starting up the project. So as you said, we keep the dynamic aspects that we have today and make only part of the configuration restricted.

However, this proposal (both yours and mine) does not solve the issue of when you are using a dynamic value to a compile time configuration. For example:

on_boot do
  config :ui, MyApp.Endpoint, some_compile_time_config: System.get_env(...)
end

Both runtime_config.exs and on_boot will gladly set this and it will have no effect. This issue also exists with the initial proposal too. In a nutshell, we can’t fully solve this issue without introducing a declarative way to describe configuration, which is something that may be added in the future, but definitely out of scope for now. However, we can consider introducing on_boot instead of the current proposal.

There are different problems to solve and while it can be tempting to solve all of them right now, gradual improvements will allow us to iterate and reap benefits on every release.


#17

@sasajuric I think there’s one fundamental thing we don’t agree on when looking at the mix config.

I don’t agree with the premise that mix config is for operator configuration - it is most definitely not. Things that are usually operator configuration - API keys, component addresses should not go in there. But things such as configuring rendering or names of components, etc, should be there. That’s what allows us to have loosely coupled components that are joined through configuration (great example are pubsub name, etc in the phoenix endpoint configuration) - this is a good thing.

I agree with what you said that we violate this for convenience - e.g. we set Ecto database credentials in the config files or phoenix port number. My view, though, is that this violation goes the other way - everything else belongs to mix config files, but things that operators might want to change don’t. They are better served by environment variables or other ways (including JSON files that you mentioned).

@hubertlepicki I think the proposed change wouldn’t “fix” anything in the compile-time vs runtime confusion space. It would be equally as confusing as it is right now. I agree that’s a problem, but it is not a problem this proposal tries to solve. We should have a separate proposal for addressing this problem.

Addressing your proposal for separate runtime and compile-time config files - that’s an idea I had in the past as well. The downside is that if you want to have both per-environment (and you’d probably do want that), now you get 8 config files - the problem of what goes where and how to understand what is the final value just got even worse not better.


Rethinking app env
#18

I think his proposal does solve one issue though: which is the user intent. We have two intents here: the intent of the user of the library and the intent of the writer of the library.

Ideally we want to conciliate both intents: if the user of the library wants to configure something on boot, then we want to check that with the intent of the writer of the library: can that configuration be set on boot or is it a compile time configuration?

This proposal is not about the library intent but about the user intent. Having on_boot or runtime_config could make the user intent clearer. Although, as you said, it won’t solve the issue of setting a compile-time configuration at runtime. This issue will remain unsolved regardless of the proposals listed here.


#19

I’d be very happy to discuss this further, but I’m worried that it’s going to cause needless noise in this thread. Maybe we should move this to a separate discussion? If so, can someone please do it, because I either lack the permissions or the know-how :slight_smile:

Edit: replied here


#21

I’m in favor of all the things in this proposal. I mentioned runtime vs compile time, just because we were talking about config. It’s already an issue today, so it doesn’t have to be solved in this.

This may also be tangentially related, but there are multiple ways to provide configuration besides the ways mentioned here:

  1. Environment variables (solved for)
  2. Secrets file in config format e.g. prod.secrets.exs (solved for)
  3. Individual secrets mounted on the file system. This is common in an environment using docker where a secret is mounted as an in memory disk, which provides better security than an environment variable. (workable, but maybe not ideal)
  4. A secrets manager, like vault. Where config is leased from a manager that has the ability to invalidate secrets. (not solved for)

This could be moved to another discussion, but I would like the ability to specify another type of configuration provider that may be more suitable for the type of configuration that is being read.


#22

What are your thoughts on the proposed on_boot? /cc @sasajuric @hubertlepicki

Yup, definitely a separate discussion. If we had configuration providers, this is about improving the built-in provider. Custom providers is a separate topic and I believe @bitwalker was working on this.