One of the major differences between running your application as a release and as a Mix project is the differences in configuration. Mix evaluates the configuration right before the application starts, releases evaluates the configuration when your application is compiled.

This implies in a large mismatch of how those two environments are used. For releases, environment variables (read by System.get_env/1) need to be set when the application is compiled and such information may not be available at this point.

Ideally, we would want a release to evaluate the configurations files in config when the release starts. One approach would be to copy the configuration files as is to the release but that’s hard to achieve in practice for two reasons:

1. A config file may import other config files and often importing those files happen dynamically. For example: import_config "#{Mix.env()}.exs". The dynamic import makes it hard for release tools to know which configuration files must be copied to a release, especially in cases like umbrella projects, where a developer may load configuration across projects

2. Even we copy today’s configuration files to a release, those configuration files rely on Mix, which is a build tool and therefore it is not available during releases

To solve those issues, we need to make sure we can discover all imports of a configuration file without evaluating its contents. We also need to introduce a new module for configuration that does not depend on Mix.

This is the goal of this proposal.

Application.Config

This proposal is about introducing a module named Application.Config. It will work similarly to the existing Mix.Config, except it belongs to the :elixir application instead of :mix. This allows releases to leverage configuration without depending on Mix.

The user API of Application.Config is quite similar to Mix.Config. There is config/2 and config/3 to define configurations. There still is import_config/1 to import new configuration files with one important difference: the argument to import_config/1 must be a literal string. So interpolation, variables or any other dynamic pattern is no longer allowed.

In order to help with configuration management, we will introduce a project option in your mix.exs, named :config_paths to help manage multiple required and optional configuration files.

In the next section we will provide an example of how configuration files used by projects like Nerves and Phoenix will have to be rewritten and then we will discuss how integration with release tools such as distillery will work.

A common example

Projects like Nerves and Phoenix generate files with built-in multi-environment configuration. Today, this configuration has an entry point config/config.exs file that imports an environment specific configuration at the bottom:

# config/config.exs
use Mix.Config

config :my_app, :some_shared_configuration, ...

import_config "#{Mix.env()}.exs"

And then each config/{dev,test,prod}.exs provides environment specific configuration. For instance:

# config/dev.exs
use Mix.Config

config :my_app, :some_dev_configuration, ...

The issue in the example above is the use of dynamic imports, such as import_config "#{Mix.env()}.exs". We will address this by defining both config/config.exs and config/#{Mix.env()}.exs as configuration entry points in your mix.exs:

# mix.exs
def project do
  [
    ...,
    config_paths: ~w(config/config.exs config/#{Mix.env()}.exs),
    ...
  ]
end

And now we can define those configuration files without dynamic imports:

# config/config.exs
import Application.Config

config :my_app, :some_shared_configuration, ...

# config/dev.exs
import Application.Config

config :my_app, :some_dev_configuration, ...

In Phoenix, the config/prod.exs case may link to a separate prod.secret.exs file. While we could also refer to this file in the :config_paths configuration in the mix.exs file, because it is only specific to production, it is more straight-forward to continue importing it at the bottom. So a config/prod.exs would look like this:

# config/prod.exs
import Application.Config

config :my_app, :some_prod_configuration, ...

import_config "prod.secret.exs"

By adding :config_paths, we are able to move the dynamic configuration to the mix.exs file and make the order that configuration files are loaded clearer.

A FarmBot example

Nerves projects tend to rely extensively on configuration files. So let’s look into existing open source Nerves projects and see how this proposal will fare. Let’s take a look at FarmBot v6.4.1.

The questions we want to answer are: if we move the FarmBot project to the proposed Application.Config, will they be able to express of all the existing idioms they do today? And, even further, will their configuration files become simpler or more complex?

From looking at its config/config.exs, we can already see a pattern that won’t work in releases: the use of Mix.env and Mix.Project.config.

We can see those variables are used to dynamically import configuration, which Application.Config won’t allow.

Those idioms are perfectly fine with how configurations work in Mix today. But they will no longer with a release built on top of Application.Config.

The solution is to move all of those imports to the :config_paths option in mix.exs. However, note that some of those dynamic imports are optional, so we will also need the ability to explicitly tag them as such:

# farmbot/mix.exs
def project do
  [
    ...,
    config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++
                   optional_config_paths(@target, Mix.env()) 
    ...
  ]
end

defp optional_config_paths("host", env),
  do: [{:optional, "config/host/#{env}.exs"}]

defp optional_config_paths(target, env),
  do: [{:optional, "config/target/#{env}.exs"}, {:optional, "config/target/#{target}.exs"}]

We believe this approach is an improvement to the previous one because it allows all environment and target specific handling to remain in the mix.exs file and not scattered around multiple configuration files.

Using it in releases

In the previous sections, we have outlined Application.Config which no longer depends on Mix and has a restricted import_config.

Now that we are able to see all of the configuration files that affect our system, a release tool, such as distillery, should be able to traverse all of those configuration files and merge them into a final config/release.exs that will be part of your release. In fact, Elixir will provide a convenient API that performs such operation, streamlining the release assembling process.

Unresolved topics

There are two important topics that we have not included in this proposal and they will be discussed in a further step.

1. What about umbrella projects? Umbrella projects also rely on configuration and we need to make sure the listed mechanisms also work well with umbrellas.

2. How to avoid common pitfalls? Even though we will migrate to Application.Config, there is nothing stopping a developer from accessing Mix (and the module defined in the mix.exs file) from their new config files. As we have seen, this may lead to errors when running releases, as releases do not have Mix available. To address this, we may introduce checks when assembling releases that make sure Mix is not invoked in configuration files, raising appropriate error messages in case they do.

Summing up

We propose a new Application.Config module and a new :config_paths project option that allows release tools to discover all of the relevant configurations in a system. Release tools can then merge and copy those configuration into releases and execute them as part of the release process, allowing dynamic calls such as System.get_env/1 to work in development and in production transparently, with or without releases.