Proposal: moving towards discoverable config files

Elixir News & Announcements Official Proposals
,
83

Precisely indeed.

I’m not saying make it harder to ‘define’ configurations for a given stage, but rather it should be made harder to ‘use’ a configuration from the wrong stage. The onus should be on the library author to do it right, not the library user.

This this this right here!

Exactly! Operator configurations belong in the system environment, not internal to a baked release application or so (thus system environment, external json, JVM’ish XML, whatever…).

I would say yes, or rather have a runtime configuration delegate to the build-time configuration if not otherwise specified (thus giving a nice fallback order).

An issue here though is that the config scripts dump into the application environment regardless, mixing up build time and run time configurations, which is why it is so easy to mis-use one that should, for example, be a run-time config at compile-time (like say a port number), this is because the interface to access both are identical when their stages are not the same.

Yes, definitely this, the interface needs to be different to get each stage’s configurations.

And honestly even run-time should not be able to access compile-time’s configurations either (just like how mix is not available at run-time, or not supposed to be anyway).

I actually edit this file fairly often to change boot-up configs… >.>

Same here, and the fact that almost no library responds to change made here is maddening, I have to kill entire process trees to get them to update (if they even update, which they won’t unless they do stupid things like baking it into the source by accessing the config at compile-time for something that should be run-time) or I have to restart the whole VM (which is when I just edit sys.config then…).

I do have complaints here, even something as simple as my discord bot elixir server has hard-coded configs that I have to change at times because of mis-designs in my dependencies that I have to deal with…

And yet you ‘can’ do things like change the purge level configuration and people have been surprised that it doesn’t work until they recompile.

1 Like
84

Yes, this is what I was talking about with sys.config.

I meant it to be “readonly” just for users who are not familiar with releases as this might be a foreign concept for them and treated as transpiled build time mix config, fallback from runtime config as you mentioned. But of course if you know what you are doing, it is are editable and perfectly fine. If they want to do runtime config (and probably should) they need explicitly design it, ideally with the support of stdlib.

The only thing to state clear to Elixir developers is when building a release, mix config is compile time and any dynamic code will be evaluated at compile time. I don’t see it as an issue to be honest. A similar problem exists with module attributes, they are compile time and as far as I know we don’t try to fix them but rather educate people:

“Every time an attribute is read inside a function, a snapshot of its current value is taken. In other words, the value is read at compilation time and not at runtime. As we are going to see, this also makes attributes useful to be used as storage during module compilation.”
Source: https://elixir-lang.org/getting-started/module-attributes.html

2 Likes
85

I deploy releases to immutable servers populated by Ansible scripts that use a templated version of sys.config for the deployed release, so I mostly care about ensuring that the boot-time interpretation of sys.config remains sane (and I never use environment variable interpolation there). Beyond that, I want to figure out how to enable Consul, Vault, and similar technologies for runtime reconfiguration of, say, Ecto. (If we decide to cycle the database credentials for one of our Repos, how do we reconfigure and restart?)

1 Like
86

There is no chance of a breaking change. We can introduce a whole new way, but we would need a backwards compatible migration path. Breaking changes only on v2.0 (which is not coming any time soon).

I know this is a long discussion but we have debated those exact points before: I don’t think we should punish how users configure libraries because some libraries are forcing users to use the application environment. There are legitimate cases of both runtime and compile-time application configuration and those should not be made harder to deal with because of a third party misuse.

However, this proposal does not concern library authors at all and they won’t see any changes for now. But something needs to be done though in how the application environment is used by libraries, for the reasons you said.

Yep, that will probably what will end-up happening, although it is a separate discussion.

I think I am missing the context here. How and where would the wiring of the system environment/external json/etc with the system happen?

Well, this is a separate issue and one that also exists in Erlang (it is a orthogonal discussion) which is that some configuration is read when the application is starting and changing this configuration won’t have any effect on the system unless the application or part of the supervision tree is restarted.

1 Like
87

One thing I think I’d really find useful, and perhaps it could be added as a part of implementing on_boot is the support for a custom boot callback. So something like:

# in mix.exs

def project do
  [
    # ...,

    boot: {MyMod, some_arg}
  ]
end

Which would cause e.g. MyMod.boot(some_arg) to be invoked after Elixir app has been started, but before any other app. This would allow me to e.g. set app env for logger at runtime from e.g. OS env.

Note that I’m not suggesting this as a replacement for on_boot, but more like a lower level interface for those of us who like to have more control, and prefer explicit code over out-of-band scripts.

Going further, I think it would be interesting if there was a before_boot callback, invoked before each app is being started.

Finally, it would be great if we could somehow affect the app starting order, so e.g. I could specify that etcd client should be started before the logger (which can obviously work only if etcd client isn’t depending on logger).

The last two features would allow me to configure logger via etcd, or generally speaking any app by using functions from another app.

Thoughts?

7 Likes
88

I assumed you could just call that module and function from on_boot like:

on_boot do
  config :some, :config
  ...
  MyMod.some_call(some_args)
end

But I agree that I already planed to abuse it this way in my head, so if the above would be considered abuse a separate callback would do as well. Either one would work for me.

Currently we’re doing a distillery hooks, that are obviously only executed when in release, and this results in some doubled logic.

2 Likes
89

I like this idea a lot although having to insert order between dependencies is a big concern. The other concern is that, if somebody calls Application.start/2 elsewhere during before_boot, it will bypass the before_boot for the started app too.

I wonder if we can slightly flip it around and make the before_boot callback per application? So you would have a boot module and in there you can do:

def before_boot(:logger) do
  {:ok, logger_config}
end

This may help us know exactly what are configuring and therefore help us detect the cases where before_boot has been rendered useless but I need to put more thought into this.

The only issue I can think for now of breaking the before_boot callback per application is that you don’t want to access etcd per invocation of the before_boot callback, so you would need to access etcd once and store it somewhere, such as an agent, but that would have to be outside of a supervision tree, since the apps have not started yet, and that makes me worried.

7 Likes
90

@sasajuric Ignore what I said, @hubertlepicki is correct.

In terms of expressive power, the boot module is the same as on_boot. I could do this with on_boot:

on_boot  do 
  Application.start(:etcd)
  for {app, config} <- ETCD.config(:my_app) do
    config app, config
  end
  :ok
end

Of course having it in a module gives more structure than a script in config, but I wanted to point out that they allow/disallow the same patterns.

5 Likes
91

This would be awesome, especially for creating scripts that start clustered deployments that are usually messed up and require handling connections, staritng apps in correct order etc. etc.

92

Good point and a nice trick, I love it :slight_smile:

Yes, I didn’t envision for this to support different patterns than on_boot. It would be nice to have a bit of more structure, but I guess what @hubertlepicki suggests is a solid suggestion which doesn’t require any additional mechanisms, other than on_boot. I could just delegate to the desired module myself, and structure my code there, or I could use your approach and provide the app boot parameters in on_boot. I think I’d be fine with either.

1 Like
93

Would on_boot have some mechanism to check the return value, or is it just fire-and-forget? For example, would there be a way to signal that the etcd call failed for some reason and should be tried again later? What if something in on_boot raises an exception?

I assume this would cause the BEAM to just shut down, but maybe it could keep going and end up failing later only if your application was designed to crash without valid configuration at boot time. In which case, it makes me wonder what on_boot is buying us, if we still have to put code inside the application for the case where on_boot fails.

1 Like
94

It all depends on the stage, I alluded to it in a prior post among the noise but essentially there would be stages that could all be handled by the same module but different functions are allowed only at different times with appropriate fallbacks to prior stages if warranted, that module would perform lookups via whatever plugins, with perhaps pre-built ones for things like the current config style or static configurations or environment variables or whatever else makes sense for standard, and people could plug in things for things like database lookups and more. In addition it could really use a registration callback/message functionality for when run-time configs change, if this is not build in then third-party libraries will continue not doing anything on updates.

95

Thanks everyone for your feedback! :heart:

We have learned a couple things from this discussion:

  1. It is probably simpler to tackle the introduction of Application.Config as a separate proposal as Application.Config has in itself the goal of decoupling configuration Mix, regardless of features that may be added or remove from the configuration API in the process

  2. The proposal, as originally written, is not enough. Most agree that we need a more explicit mechanism to separate runtime/compile-time configuration. Although we do not necessarily agree on how hard this separation has to be or its API

There are also more operational concerns regarding the semantics of a chosen solution. What happens if the on_boot callback fails? Can we do something about it? This is important on Nerves devices as you usually don’t want failing to boot to bring the VM down.

For now, I have decided to hold on moving this proposal forward. The reason is that, before we have minimal releases in Elixir itself, we won’t be sure on how to answer some of those questions. It may also be that, by adding releases to Elixir itself may solve some of those problems. For example, what if, once we add releases to Elixir, it becomes impossible to run MIX_ENV=prod mix run by default and instead it points you to a release? If this happens, the distinction between Mix and Releases is completely gone, at least for the prod environment. We are speculating here but the point still stands: without releases in Elixir, we cannot be 100% sure the direction we choose is the best way to go. So we will proceed on adding releases to Elixir and then hopefully this discussion can be brought back to life.

Once again, thanks for all the feedback. It was a fantastic discussion, as always.

29 Likes
96

Adding releases in Elixir is the perfect way to know where you stand before making important decisions which may make the future migration even harder.

Good call! :023:

3 Likes
97

Just putting a couple of belated thoughts into this thread for when we eventually circle back to it.

  1. My perception is that newcomers to the BEAM often miss the important point that there are compile-time settings at all. In particular, I’ve seen people try to ship API tokens and rabbitmq queue names using module attributes, and I’ve seen similar errors propagated in Mix.config. These fail to work well in a multi-node deployment scenario, and also it often causes problems during compilation phase when mix configs are changed but the dependency module itself doesn’t get recompiled as expected. There needs to be a clear separation between compile time and runtime configs.

  2. In practice I’ve found that I build releases via distillery, injecting a fake sys.config & vm.args into a single deployable package (like a container), and then use ansible to set those config files as required for runtime. This split between safe code and unsafe credentials is nicely enforced, but its in erlang format and IMO a bit too high a burden for the general developer-sysadmin combo one usually works with.

  3. Don’t under-estimate the value of being able to query and set at runtime via Application.get|set... parameters. I use this for tuning a cache + API server that processes 80k transactions/hour where I don’t want to restart the VM and lose all that cached goodness. This is used to tweak supervisor and worker restart values, runtime tunables like http connection longevity and load balancer connections.

  4. Purely for me, I find there are too many places and files to configure. A typical application has 10+ dependencies, some internal, some external, each with its own expectations of what goes where, and what’s exposed to users. I expect this is very intimidating for newcomers just trying to get that first application up and running.

  5. I’m now a huge fan of passing runtime configuration, such as secrets and endpoints, through via tools like etcd and vault. This, in practice, means having a configuration system in elixir, that allows delegation to some other external system for retrieving parameters, that is independent of whatever a given module defines - callbacks, if you will. Most of these values will have a lifetime or expiration associated with them, and I’d like to kill off my workers when their tokens have expired, or when the configuration changes during runtime. Most of these things will be inside my own apps, so its reasonable to expect I own configuration and dealing with changes, but I’d at least like the config system to be able to handle this use case with me.

  6. I would like to see, in the BEAM runtime, not in elixir, a standard way of retrieving a value from the App Env if set, or failing that, a given environment variable, and failing that, the sys.config file. Many of us run into this problem: rebar, rebar3, mix, relx, erlang.mk, and even emake for dependencies. The whole community benefits from a consistent way of handling these. This would then slowly be used consistently throughout the BEAM ecosystem. Of course, I’d really like to be able to write native elixir syntax in this file though, so not sure if I can have my cake and eat it too.

  7. The reasoning for the above ordering is that these should roughly fall into:

    • App env: fast lookup inside the runtime – you deliberately set this, or its already been retrieved from a lower/slower level and cached in the App env ETS tables for speed. I generally can afford retrieving this value every time a new worker or significant function in a gen_* is called.
    • environment variable – secret data – if you passed this through, you’re probably trying to inject a secret token into the app at runtime. So let’s keep it secret, and not actually write these out to an intermediate file when using REPLACE_OS_VARS which is a bit of a shock the first time you see it
    • sys.config: file lookups, or whatever the individual module/app thinks is a reasonable default. If you’ve gotten this far, the value is not already configured at a higher layer, and nor is it a secret nor dynamic. So give whatever we have lying around on disk, and assume it is a sensible default, and let it be cached for later.

Many thanks to the noble discussion above, I’ll need a couple more days to digest all the ideas people have put forth!

4 Likes
98

That was a lengthy discussion!

My 3 cents. I love the initial Application.Config proposal. I felt like there is a circular dependency. Mix produces config depending on env. Config uses mix to import files. It gently pushes towards not using Mix in configs which is excellent for releases.

I’m not sure about the on_boot proposal.
When I was starting with Phoenix, I wanted to configure port using ENV, and run two copies of my project to check some Erlang distributed. I compiled it once, copied to another folder and changed the environment variable. Of course, the port number was in the compiled artefact so that I couldn’t run the second instance.

When consulting at Bleacher Report, they had a similar problem. They wanted to build the binary once and then configure it via env variables. I believe that was even before introducing {:system, "ENV_VAR"}.

My point is: users will still need to learn to put this stuff in right places, and on_boot isn’t much different for me from runtime.prod.exs, runtime.dev.exs. You still need to know where to put stuff, so it works as expected.

Newcomers need to learn how macros and compile-time evaluation work to configure the app. They also need to learn how library creator uses it. Is the setting read on each invocation or only during startup? If I modify port in the app env, will the app start accepting new connections on a different port or do I need to restart it?

My dream config would have three sections:

  • stuff that lands in the binary with warnings in comments that reading from external sources might not work as expected
  • stuff that is evaluated when the application starts where you probably want to read from files or external sources
  • stuff that is evaluated on demand with a warning that costly operations are discouraged, so don’t read files here, read application environment.

Then in the library code, one would invoke:

  • Config.get_static_value
  • Config.get_startup_value
  • Config.get_on_demand_value

With this approach, we could check at compile time how library uses the config and generate warnings like:
“You are setting port value in on_demand section, but the library reads it only using get_startup_value. Consider putting the config in startup section”.

The downside is that it would encourage even more bloat in configs.

5 Likes
99

In my current elixir applications I always resolve configuration at runtime with simpel function wrappers around Application.get_env. I mainly do this since I also use conform to have a human readable configuration file our Devops guy can provision and create using Chef / Ansible / … It also allows me to change my configuration file and dynamically apply it to my running application!

Another thing I would like to mention is that I always see two kinds of configuration:

  • Glue configuration: This is configuration needed to stitch up your dependencies with your app (What is your phoenix endpoint), but also includes mocks and behaviour definition. This type of configuration is mostly done by the developer and never changed
  • Actual Configuration: DB passwords, http ports, secrets, keys, urls to other apps… this type of configuration is mostly done by devops teams

I would love to see a seperation between Glue configuration / actual configuration, just my two cents on the matter

100

I am going to resurrect this subject unless you mind as there are new things that just now popped up on my horizon.

@bitwalker released Distillery 2.0 (https://dockyard.com/blog/2018/08/23/announcing-distillery-2-0) and it does tackle the issue of run-time configuration in rather elegant way.

They introduced concept of Config Providers (https://hexdocs.pm/distillery/2.0.8/config/runtime.html#config-providers) and you can actually write your own (https://hexdocs.pm/distillery/2.0.8/extensibility/config_providers.html).

There is a Mix.Config provider that is dead easy to use. You just create rel/config/config.exs file with extracted all the runtime configuration that you want to take from say environment variables. You can execute Elixir code there safely, which means you can do System.get_env("PORT") |> String.to_integer(), which was a pain to do with REPLACE_OS_VARS solution it used before.

Moreover, this config file does not have to be part of the release at all and you can just dump it on the server, having all the secrets directly inserted in it and it will never even sit in source code repository.

I like it very much. Not sure what your thoughts are @josevalim and the rest of Elixir team, but it does solve the runtime configuration issues for my use cases completely.

10 Likes
101

Thanks @hubertlepicki for the follow-up.

One of the conclusions of this conversation was precisely “Most agree that we need a more explicit mechanism to separate runtime/compile-time configuration”. So I believe the direction Distillery 2.0 took of providing an explicit runtime configuration for releases is very much inline with the thoughts of the core team and of the community.

Excellent work from @bitwalker!

11 Likes
102

I hope I’m not repeating an existing idea, but this seems solvable by building a core component in the style of the Logger?

  • Build a simple key-value database which is initially populated from a variety of configurable sources (ideally make this pluggable so people can dream up new things in the future)

  • Boot an instance of this config at the pre-boot point discussed above

  • API to access config is standardised, but the normal usage will be to access this global config instance

  • However, it should be possible to create additional config instances to pass to libraries if required. This makes it straightforward to standardise on a config standard, set defaults, etc, but for example if you need your favourite twitter library to access multiple user accounts you would simply instantiate multiple config instances and customise for each user.

This would appear to solve the issue of both general default configurations, potentially merged from a variety of sources, plus the ability to generate customised config for particular instances of some module. There is also a standardised way to pass config to modules (either global or with the option for local override). In both cases of system default and customised instance you continue to have the ability to make runtime changes to the config.

This proposal clearly doesn’t touch compile time config, but I think that’s adequately covered already. I am not clear what language options would be sufficiently fast for putting such a system config option in a hot code path? Perhaps ETS tables?