It all depends on the stage, I alluded to it in a prior post among the noise but essentially there would be stages that could all be handled by the same module but different functions are allowed only at different times with appropriate fallbacks to prior stages if warranted, that module would perform lookups via whatever plugins, with perhaps pre-built ones for things like the current config style or static configurations or environment variables or whatever else makes sense for standard, and people could plug in things for things like database lookups and more. In addition it could really use a registration callback/message functionality for when run-time configs change, if this is not build in then third-party libraries will continue not doing anything on updates.
OvermindDL1
josevalim
Thanks everyone for your feedback! 
We have learned a couple things from this discussion:
-
It is probably simpler to tackle the introduction of
Application.Configas a separate proposal asApplication.Confighas in itself the goal of decoupling configuration Mix, regardless of features that may be added or remove from the configuration API in the process -
The proposal, as originally written, is not enough. Most agree that we need a more explicit mechanism to separate runtime/compile-time configuration. Although we do not necessarily agree on how hard this separation has to be or its API
There are also more operational concerns regarding the semantics of a chosen solution. What happens if the on_boot callback fails? Can we do something about it? This is important on Nerves devices as you usually don’t want failing to boot to bring the VM down.
For now, I have decided to hold on moving this proposal forward. The reason is that, before we have minimal releases in Elixir itself, we won’t be sure on how to answer some of those questions. It may also be that, by adding releases to Elixir itself may solve some of those problems. For example, what if, once we add releases to Elixir, it becomes impossible to run MIX_ENV=prod mix run by default and instead it points you to a release? If this happens, the distinction between Mix and Releases is completely gone, at least for the prod environment. We are speculating here but the point still stands: without releases in Elixir, we cannot be 100% sure the direction we choose is the best way to go. So we will proceed on adding releases to Elixir and then hopefully this discussion can be brought back to life.
Once again, thanks for all the feedback. It was a fantastic discussion, as always.
dimitarvp
Adding releases in Elixir is the perfect way to know where you stand before making important decisions which may make the future migration even harder.
Good call! 
dch
Just putting a couple of belated thoughts into this thread for when we eventually circle back to it.
-
My perception is that newcomers to the BEAM often miss the important point that there are compile-time settings at all. In particular, I’ve seen people try to ship API tokens and rabbitmq queue names using module attributes, and I’ve seen similar errors propagated in Mix.config. These fail to work well in a multi-node deployment scenario, and also it often causes problems during compilation phase when mix configs are changed but the dependency module itself doesn’t get recompiled as expected. There needs to be a clear separation between compile time and runtime configs.
-
In practice I’ve found that I build releases via distillery, injecting a fake sys.config & vm.args into a single deployable package (like a container), and then use ansible to set those config files as required for runtime. This split between safe code and unsafe credentials is nicely enforced, but its in erlang format and IMO a bit too high a burden for the general developer-sysadmin combo one usually works with.
-
Don’t under-estimate the value of being able to query and set at runtime via
Application.get|set...parameters. I use this for tuning a cache + API server that processes 80k transactions/hour where I don’t want to restart the VM and lose all that cached goodness. This is used to tweak supervisor and worker restart values, runtime tunables like http connection longevity and load balancer connections. -
Purely for me, I find there are too many places and files to configure. A typical application has 10+ dependencies, some internal, some external, each with its own expectations of what goes where, and what’s exposed to users. I expect this is very intimidating for newcomers just trying to get that first application up and running.
-
I’m now a huge fan of passing runtime configuration, such as secrets and endpoints, through via tools like
etcdandvault. This, in practice, means having a configuration system in elixir, that allows delegation to some other external system for retrieving parameters, that is independent of whatever a given module defines - callbacks, if you will. Most of these values will have a lifetime or expiration associated with them, and I’d like to kill off my workers when their tokens have expired, or when the configuration changes during runtime. Most of these things will be inside my own apps, so its reasonable to expect I own configuration and dealing with changes, but I’d at least like the config system to be able to handle this use case with me. -
I would like to see, in the BEAM runtime, not in elixir, a standard way of retrieving a value from the App Env if set, or failing that, a given
environment variable, and failing that, thesys.configfile. Many of us run into this problem: rebar, rebar3, mix, relx, erlang.mk, and even emake for dependencies. The whole community benefits from a consistent way of handling these. This would then slowly be used consistently throughout the BEAM ecosystem. Of course, I’d really like to be able to write native elixir syntax in this file though, so not sure if I can have my cake and eat it too. -
The reasoning for the above ordering is that these should roughly fall into:
- App env: fast lookup inside the runtime – you deliberately set this, or its already been retrieved from a lower/slower level and cached in the App env ETS tables for speed. I generally can afford retrieving this value every time a new worker or significant function in a gen_* is called.
- environment variable – secret data – if you passed this through, you’re probably trying to inject a secret token into the app at runtime. So let’s keep it secret, and not actually write these out to an intermediate file when using
REPLACE_OS_VARSwhich is a bit of a shock the first time you see it sys.config: file lookups, or whatever the individual module/app thinks is a reasonable default. If you’ve gotten this far, the value is not already configured at a higher layer, and nor is it a secret nor dynamic. So give whatever we have lying around on disk, and assume it is a sensible default, and let it be cached for later.
Many thanks to the noble discussion above, I’ll need a couple more days to digest all the ideas people have put forth!
tomekowal
That was a lengthy discussion!
My 3 cents. I love the initial Application.Config proposal. I felt like there is a circular dependency. Mix produces config depending on env. Config uses mix to import files. It gently pushes towards not using Mix in configs which is excellent for releases.
I’m not sure about the on_boot proposal.
When I was starting with Phoenix, I wanted to configure port using ENV, and run two copies of my project to check some Erlang distributed. I compiled it once, copied to another folder and changed the environment variable. Of course, the port number was in the compiled artefact so that I couldn’t run the second instance.
When consulting at Bleacher Report, they had a similar problem. They wanted to build the binary once and then configure it via env variables. I believe that was even before introducing {:system, "ENV_VAR"}.
My point is: users will still need to learn to put this stuff in right places, and on_boot isn’t much different for me from runtime.prod.exs, runtime.dev.exs. You still need to know where to put stuff, so it works as expected.
Newcomers need to learn how macros and compile-time evaluation work to configure the app. They also need to learn how library creator uses it. Is the setting read on each invocation or only during startup? If I modify port in the app env, will the app start accepting new connections on a different port or do I need to restart it?
My dream config would have three sections:
- stuff that lands in the binary with warnings in comments that reading from external sources might not work as expected
- stuff that is evaluated when the application starts where you probably want to read from files or external sources
- stuff that is evaluated on demand with a warning that costly operations are discouraged, so don’t read files here, read application environment.
Then in the library code, one would invoke:
Config.get_static_valueConfig.get_startup_valueConfig.get_on_demand_value
With this approach, we could check at compile time how library uses the config and generate warnings like:
“You are setting port value in on_demand section, but the library reads it only using get_startup_value. Consider putting the config in startup section”.
The downside is that it would encourage even more bloat in configs.
JanStevens
In my current elixir applications I always resolve configuration at runtime with simpel function wrappers around Application.get_env. I mainly do this since I also use conform to have a human readable configuration file our Devops guy can provision and create using Chef / Ansible / … It also allows me to change my configuration file and dynamically apply it to my running application!
Another thing I would like to mention is that I always see two kinds of configuration:
- Glue configuration: This is configuration needed to stitch up your dependencies with your app (What is your phoenix endpoint), but also includes mocks and behaviour definition. This type of configuration is mostly done by the developer and never changed
- Actual Configuration: DB passwords, http ports, secrets, keys, urls to other apps… this type of configuration is mostly done by devops teams
I would love to see a seperation between Glue configuration / actual configuration, just my two cents on the matter
hubertlepicki
I am going to resurrect this subject unless you mind as there are new things that just now popped up on my horizon.
@bitwalker released Distillery 2.0 (Announcing Distillery 2.0 - DockYard) and it does tackle the issue of run-time configuration in rather elegant way.
They introduced concept of Config Providers (Runtime Configuration - Distillery Documentation) and you can actually write your own (Config Providers - Distillery Documentation).
There is a Mix.Config provider that is dead easy to use. You just create rel/config/config.exs file with extracted all the runtime configuration that you want to take from say environment variables. You can execute Elixir code there safely, which means you can do System.get_env("PORT") |> String.to_integer(), which was a pain to do with REPLACE_OS_VARS solution it used before.
Moreover, this config file does not have to be part of the release at all and you can just dump it on the server, having all the secrets directly inserted in it and it will never even sit in source code repository.
I like it very much. Not sure what your thoughts are @josevalim and the rest of Elixir team, but it does solve the runtime configuration issues for my use cases completely.
josevalim
Thanks @hubertlepicki for the follow-up.
One of the conclusions of this conversation was precisely “Most agree that we need a more explicit mechanism to separate runtime/compile-time configuration”. So I believe the direction Distillery 2.0 took of providing an explicit runtime configuration for releases is very much inline with the thoughts of the core team and of the community.
Excellent work from @bitwalker!
edwildgoose
I hope I’m not repeating an existing idea, but this seems solvable by building a core component in the style of the Logger?
-
Build a simple key-value database which is initially populated from a variety of configurable sources (ideally make this pluggable so people can dream up new things in the future)
-
Boot an instance of this config at the pre-boot point discussed above
-
API to access config is standardised, but the normal usage will be to access this global config instance
-
However, it should be possible to create additional config instances to pass to libraries if required. This makes it straightforward to standardise on a config standard, set defaults, etc, but for example if you need your favourite twitter library to access multiple user accounts you would simply instantiate multiple config instances and customise for each user.
This would appear to solve the issue of both general default configurations, potentially merged from a variety of sources, plus the ability to generate customised config for particular instances of some module. There is also a standardised way to pass config to modules (either global or with the option for local override). In both cases of system default and customised instance you continue to have the ability to make runtime changes to the config.
This proposal clearly doesn’t touch compile time config, but I think that’s adequately covered already. I am not clear what language options would be sufficiently fast for putting such a system config option in a hot code path? Perhaps ETS tables?
