One of the major differences between running your application as a release and as a Mix project is the differences in configuration. Mix evaluates the configuration right before the application starts, releases evaluates the configuration when your application is compiled.
This implies in a large mismatch of how those two environments are used. For releases, environment variables (read by System.get_env/1
) need to be set when the application is compiled and such information may not be available at this point.
Ideally, we would want a release to evaluate the configurations files in config
when the release starts. One approach would be to copy the configuration files as is to the release but thatâs hard to achieve in practice for two reasons:
-
A config file may import other config files and often importing those files happen dynamically. For example:
import_config "#{Mix.env()}.exs"
. The dynamic import makes it hard for release tools to know which configuration files must be copied to a release, especially in cases like umbrella projects, where a developer may load configuration across projects -
Even we copy todayâs configuration files to a release, those configuration files rely on
Mix
, which is a build tool and therefore it is not available during releases
To solve those issues, we need to make sure we can discover all imports of a configuration file without evaluating its contents. We also need to introduce a new module for configuration that does not depend on Mix.
This is the goal of this proposal.
Application.Config
This proposal is about introducing a module named Application.Config
. It will work similarly to the existing Mix.Config
, except it belongs to the :elixir
application instead of :mix
. This allows releases to leverage configuration without depending on Mix.
The user API of Application.Config
is quite similar to Mix.Config
. There is config/2
and config/3
to define configurations. There still is import_config/1
to import new configuration files with one important difference: the argument to import_config/1
must be a literal string. So interpolation, variables or any other dynamic pattern is no longer allowed.
In order to help with configuration management, we will introduce a project option in your mix.exs
, named :config_paths
to help manage multiple required and optional configuration files.
In the next section we will provide an example of how configuration files used by projects like Nerves and Phoenix will have to be rewritten and then we will discuss how integration with release tools such as distillery will work.
A common example
Projects like Nerves and Phoenix generate files with built-in multi-environment configuration. Today, this configuration has an entry point config/config.exs
file that imports an environment specific configuration at the bottom:
# config/config.exs
use Mix.Config
config :my_app, :some_shared_configuration, ...
import_config "#{Mix.env()}.exs"
And then each config/{dev,test,prod}.exs
provides environment specific configuration. For instance:
# config/dev.exs
use Mix.Config
config :my_app, :some_dev_configuration, ...
The issue in the example above is the use of dynamic imports, such as import_config "#{Mix.env()}.exs"
. We will address this by defining both config/config.exs
and config/#{Mix.env()}.exs
as configuration entry points in your mix.exs
:
# mix.exs
def project do
[
...,
config_paths: ~w(config/config.exs config/#{Mix.env()}.exs),
...
]
end
And now we can define those configuration files without dynamic imports:
# config/config.exs
import Application.Config
config :my_app, :some_shared_configuration, ...
# config/dev.exs
import Application.Config
config :my_app, :some_dev_configuration, ...
In Phoenix, the config/prod.exs
case may link to a separate prod.secret.exs
file. While we could also refer to this file in the :config_paths
configuration in the mix.exs
file, because it is only specific to production, it is more straight-forward to continue importing it at the bottom. So a config/prod.exs
would look like this:
# config/prod.exs
import Application.Config
config :my_app, :some_prod_configuration, ...
import_config "prod.secret.exs"
By adding :config_paths
, we are able to move the dynamic configuration to the mix.exs
file and make the order that configuration files are loaded clearer.
A FarmBot example
Nerves projects tend to rely extensively on configuration files. So letâs look into existing open source Nerves projects and see how this proposal will fare. Letâs take a look at FarmBot v6.4.1.
The questions we want to answer are: if we move the FarmBot project to the proposed Application.Config
, will they be able to express of all the existing idioms they do today? And, even further, will their configuration files become simpler or more complex?
From looking at its config/config.exs, we can already see a pattern that wonât work in releases: the use of Mix.env
and Mix.Project.config
.
We can see those variables are used to dynamically import configuration, which Application.Config
wonât allow.
Those idioms are perfectly fine with how configurations work in Mix today. But they will no longer with a release built on top of Application.Config
.
The solution is to move all of those imports to the :config_paths
option in mix.exs
. However, note that some of those dynamic imports are optional, so we will also need the ability to explicitly tag them as such:
# farmbot/mix.exs
def project do
[
...,
config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++
optional_config_paths(@target, Mix.env())
...
]
end
defp optional_config_paths("host", env),
do: [{:optional, "config/host/#{env}.exs"}]
defp optional_config_paths(target, env),
do: [{:optional, "config/target/#{env}.exs"}, {:optional, "config/target/#{target}.exs"}]
We believe this approach is an improvement to the previous one because it allows all environment and target specific handling to remain in the mix.exs
file and not scattered around multiple configuration files.
Using it in releases
In the previous sections, we have outlined Application.Config
which no longer depends on Mix and has a restricted import_config
.
Now that we are able to see all of the configuration files that affect our system, a release tool, such as distillery, should be able to traverse all of those configuration files and merge them into a final config/release.exs
that will be part of your release. In fact, Elixir will provide a convenient API that performs such operation, streamlining the release assembling process.
Unresolved topics
There are two important topics that we have not included in this proposal and they will be discussed in a further step.
-
What about umbrella projects? Umbrella projects also rely on configuration and we need to make sure the listed mechanisms also work well with umbrellas.
-
How to avoid common pitfalls? Even though we will migrate to
Application.Config
, there is nothing stopping a developer from accessing Mix (and the module defined in themix.exs
file) from their new config files. As we have seen, this may lead to errors when running releases, as releases do not have Mix available. To address this, we may introduce checks when assembling releases that make sureMix
is not invoked in configuration files, raising appropriate error messages in case they do.
Summing up
We propose a new Application.Config
module and a new :config_paths
project option that allows release tools to discover all of the relevant configurations in a system. Release tools can then merge and copy those configuration into releases and execute them as part of the release process, allowing dynamic calls such as System.get_env/1
to work in development and in production transparently, with or without releases.