This is totally fine. 
Yes! Libraries should only require application configuration if strictly necessary. Our recent library guideline discusses this: https://hexdocs.pm/elixir/master/library-guidelines.html
If the user of your library wants to use the app environment, that should be their choice. Library authors shouldnât impose them.
After reading through this thread, Iâm left wondering what specific problem this proposal is trying to solve, even after itâs been summarized and re-stated several times.
Introducing Application.Config that would work similarly to Mix.Config, but be present within a release where Mix isnât available.
I havenât seen much discussion about this point, which perhaps indicates that people would be fine with it, assuming that it all behaves equivalently to the way it does today.
I personally think it would be an improvement if we didnât have config files being included dynamically, because it tends to make it difficult to figure out what the final value will be, based on several files including each other. It seems that this is more likely to be the case for Nerves-based projects than for others because we often like to work on pieces of the system on our laptops, only using Nerves to deploy the firmware to a real device at the end. That can lead to configuration file trees that pull in different files depending on whether youâre running the whole system on the device (using a release) or just running the Phoenix-based UI on your laptop (using Mix). Under the original proposal, I think it would just be a matter of putting something like this in mix.exs for eye_fw:
# Note: @target is already set normally in a Nerves-based mix.exs file
config_paths: [
"../../eye_ui/config/base.exs",
"../../eye_ui/config/prod.exs",
"config/config.exs",
"config/#{@target}"
]
This seems fine to me. It keeps things pretty explicit as far as the order in which configurations will get applied, and allows me to not have to copy/paste the contents of the config files depending on which thing Iâm trying to run.
Assuming that the configuration doesnât rely on Mix, should we apply the same config file both at compile time and runtime. I donât recall it being specifically stated, but my understanding is that this would be a new step as a release is starting up that would process and execute the configuration somehow before starting any of the Applications in the release, so that they can have their configurations readable using Application.get_env/2 exactly as they do today, but based on the run-time environment of the release.
There are a lot of opinions in the thread, and itâs unclear to me whether weâre specifically trying to:
Reduce confusion that people often have initially about whether the config.exs is applied at compile time, runtime, or both.
Patch over existing libraries that are doing the âwrong thing,â or allowing/encouraging their users to do the âwrong thing.â
Solve the question of how library authors should allow/encourage their users to specify compile-time vs run-time options and settings going forward.
For example, would this problem be solved without making any changes at all to Mix and releases if libraries consistently supported a way for the user to specify that they want to read from an environment variable at runtime (like {:system, "DB_URL"} or just "${DB_URL}") instead of Mix literally calling System.get_env("DB_URL") in the configuration script at compile time? Could we solve the problem by doing what REPLACE_OS_VARS does, only from inside the release instead of the way it works now by rewriting the release config?
Solve the question of how library users should specify settings going forward. In particular, for the common case that they want to read environment variables at run-time and not at compile-time.
Handle all the unusual cases people currently have in their config.exs just because itâs there and itâs an executable Elixir script.
Assuming that we are going to run the same configuration at compile-time and run-time, should we have a way for the user to only run certain parts of it (via on_boot) or not run certain parts of it (via on_compile).
If the current guidance for library authors is that they should avoid using application configuration, maybe we end up needing config.exs less and less because the docs for the libraries will tell them to pass options into their init callback instead of putting something in config.exs. If thatâs the future, then maybe thereâs nothing to actually change about the way it works other than libraries, documentation, and community culture.
For example, if the config.exs were only used to declare which environment variable to use and what its default value should be if not set, then it doesnât matter whether youâre running in a release or not because itâs only declaring the name of the environment variable at both compile time and run time, not its contents. It would be up to the library to substitute the value at the time it needs to.
What Iâm getting at here is that maybe we donât need a generic on_boot macro that can execute arbitrary Elixir code, if the real problem weâre trying to solve is how to declare that we want to read from an environment variable and what the default value should be if itâs not set.
It feels like weâre dealing with a leaky abstraction here.
In some cases configuration will change the code that gets compiled. e.g. A different set of macros will be run and the code generated by a compile-time configuration will have caused that. At other times, the configuration value will be used in code that doesnât change at compile time. e.g. The host to connect to for an API. In a degenerate case, the value will be inlined as a constant at compile time and you have to go and complain to the library author that the code canât be run in production⊠But thatâs what the guidelines are for.
As a user I can not be expected to know which configuration will be which of the above scenarios implicitly. The fact that the MySQL Ecto adapter and the Postgrex one produce different compile-time code is âunknowableâ to me because itâs based on how those libraries are implemented. You could do the same with log levels where the config changed macros and no-opâed out everything. Clearly changing those flags at runtime is meaningless.
I guess Iâm looking for the Principle of Least Surprise here. Having configuration that modified the code at compile time in a separate location from configuration that only effects runtime and enforced somehow seems like an ideal solution. (an exercise left to the reader. 
) It makes the contract explicit and would require the library writer to specify the type of configuration, compile time or runtime, based on the implementation.
You provided a very good summary of the opinion going around the thread. To answer your bullets:
Application.Config exists so we can also load configuration in a release, where Mix is not available. In a way, it is orthogonal to the other topics on this thread. Think of it as a replacement/alternative to conform.
The original proposal is about applying the same config file both at compile time and runtime. This would allow us to remove both {:system, "DB_URL"} and the REPLACE_OS_VARS workarounds by literally calling System.get_env("DB_URL"). However, it doesnât change any of the other bullet items you mentioned, in particular, it doesnât change how library authors should read from the application environment. It may make some cases worse, for example, unusual things being done a config.exs file will now also run in a release.
The on_boot is a half way step from the current proposal where, instead of running the whole config file in both compile-time and a release, a release will run only the on_boot parts. Everything else is assumed to run on compile-time, so there is no on_compile. The original proposal and on_boot aim to solve the same problem, but with different degrees of explicitness.
Excluding Application.Config discussion, we basically have four options:
Run config files only at compile time (this is what happens today). This leaves users empty handed about how to configure their releases
Run the same config files at both compile-time and runtime. The issue with this is that any crazy thing happening in a config.exs will now also run in production because there is no distinction what happens during compile time and what happens on boot. If you are doing something âtoo crazyâ, it may even make booting the release fail
Run config files only at compile time but allow them to declare chunks of code that should execute at runtime (on_boot). This attempts to circumvent the issues in 2 and it makes releases and Mix behave a bit closer to each other
Run config files only at compile time and introduce a configuration file that is specific to releases where you would put the runtime/on_boot configuration. The issue here is that you may end-up duplicating some of the config/prod.exs in the release_config.exs
If we are talking about leaky abstraction⊠isnât that forcing the leaky abstraction onto the users? As a user, I would love to not have to care about this. Imagine how confusing it will be if every time I want to configure an application I need to figure which one of those buckets I need to put my configuration in. The only case where a user should have to care about this distinction is when they want to configure something that happens at compile-time during runtime.
Given the following:
I donât think you can hide the context from users. If a library needs the data at compile time, I canât set it at runtime. And if the user is invoking e.g. System functions, then they might fail if executed in a wrong environment, or return a wrong result.
So I think something has to give here. Either configs are allowed in only one context (either runtime or compile time), or dynamic code is not supported (i.e. only constants can be provided), or users need to be aware of the context and choose where they want to provide the value. But I think that this is exactly what on_boot is solving, right?
Yes, exactly. There are multiple ways to solve this problem, on_boot is one of them. on_boot declares the user intent and if we allow library authors to declare what is compile-time and what is runtime, we can use this information and notify the user of any mismatches. Having separate files for runtime configs would have the same effect.
The goal with on_boot is that users do not need to know upfront what goes what. We can use the tooling to automatically detect conflicts. If I could do an analogy, forcing the users to declare what is compile vs runtime upfront would be equivalent to Git requiring me to explicitly merge all commits, even if there are no conflicts. on_boot is about bothering the user only when conflicts are detected. In order for this to work though, we need to increase the burden on library developers, which is fine and probably warranted, since most developers should rely less on compile-time application config anyway.
Another idea, which might make the difference between compile-time configuration vs run-time(/on-boot) configuration more clear to the user but only in the instances they have to care about it is to:
Application.Config which has the same behaviour as Mix.Config but lives in :elixir instead of :mix so it is available in Releases. (as in original proposal)
mix.exs file by using a :config_paths key.(as in original proposal)
mix.exs information for compile-mode or for run-time/boot-time-mode so that we can in e.g. the implementation of the mix.exs's project() fields (most importantly: :config_paths) potentially choose between different groups of configuration files. This is very similar to how different Mix environments are currently used to e.g. switch between what dependencies you include.import_config just like you do today (including string-replacements) is still allowed, but we might decide to deprecate this feature, and definitely show warnings (or potentially hard errors) when the configuration file that was linked to can not currently be found because its string replacement was âtoo smartâ, and point the user towards the new way to configure their application using a conditional config_paths: list.I think that when we would set it up like this, the abstraction is the least leaky.
Example usage:
:config_paths is constant):config_paths.:config_paths setup.In the simplest case, it would looke like:
def project do
[
...
config_paths: ~w(config/config.exs config/#{Mix.env()}.exs)
...
]
end
Expanded to do different things based on compile-time/boot-time mode:
def project do
[
...
config_paths: ~w{config/base_config.exs} ++ config_paths(Mix.env(), Mix.configuration_target())
...
]
end
def config_paths(_, :compile_time), do: ~w{config/compile_time_overrides.exs}
def config_paths(:dev, :boot_time) do: ~w{config/production_boot_overrides.exs config/production_secrets.exs}
def config_paths(:prod, :boot_time) do: ~w{config/dev_boot_overrides.exs}
This proposed solution is backwards compatible, because:
Mix.Config to Application.Config is optional (although we probably should make Application.Config the new default for new projects).Mix.Config will just like now work as it always did: Releases only can use it during compiletime, applications running through mix use it at compiletime + boot-time.Application.Config, we look at the :config_paths field in the mix.exs file. This field (and potentially other fields of the project() function) might return different results based on the current Mix.configuration_target() (Maybe we can find a better name for configuration_target?), which will allow Releases to know what files to include for boot-time, and will allow all projects to only expose the correct configuration in the correct configuration environment.Application.Config will complain if import_config is encountered inside a configuration file that is flagged for inclusion into the list of boot-time configuration. Mix.Config will do no such thing and will happily âjust not workâ on boot-time for a Release.Theres the âit worked on my machineâ problem - someone can do something wrong but it will still work until deployed - IMO this kind of potential problems must be impossible or as edge as it gets and leaving the distinction is a step in the opposite direction.
I second that.
Users in this context are programmers, users of the language, right? Then they actually have to care unless we somehow manage to completely remove the under the hood distinction on_boot solution, if adopted, already assumes that users know what it means.
Thatâs also my understanding but I think the users have to be aware of the context no matter what it is (if configs are only runtime or compile time only).
Good point. The only time they donât have to be aware of the context is if thereâs no dynamic code (i.e. only constants are provided).
Isnât this a common tradeoff that in Elixir world people tend to err on the side of explicit option e.g. Phoenix vs Rails design. Would that duplication really be that bad?
I donât think thatâs the common trade-off. Why force users to care about a library decision if that is not going to impact them? We should push those concerns to users only when there is a conflict.
Right. The question is: can we move the dynamic code concerns to on_boot (or something similar)?
We chose to solve this problem a different way.
For a while we used things like deferred_config to pull env vars into config at runtime. This worked, but there were setting that weâd like to update while the app was running. We did this by writing a library that uses configparser_ex to parse ini files and file_system to monitor a config directory for changes to ini files. When the library app starts it loads any ini files found and stores the data in an erts table. We have a github checkin hook that will update ini files in production when changes are checked in. The library gets a notification and reloads the ini file that changed and updates erts. This allows us to update config while the application is running.
We treat config/config.exs as compile time settings that donât change often (or ever.) And all other settings go into the ini files.
I would love to not have to care about it too, so we agree there. But I donât know that itâs avoidable. Either way, I guess I would rather force us to be explicit and have things work in an understandable way as opposed to things being implicit and wrong in a lot of cases. on_boot is still an explicit mechanism, just one that doesnât seem obvious to me at first glance.
I guess that is true. If thereâs a way for me to look up an item and say âyes this config option can be modified at runtimeâ, therefore I need to move it somewhere else⊠I guess, to me, on_boot (or whatever it ends up being called) or a different file doesnât make a huge difference at that point. I think that passes my test for being explicit. Be default, everything is compile time but some things are made special and become runtime (if applicable).
Iâve read over this thread, and I have some thoughts about the situation. Why canât we just keep config files as static, compile-time only configuration, and use the code for runtime config? It seems ideal to me since callbacks like init seem to serve this very purpose. That would make it very explicit as to what is configured at runtime vs. compile time, since you are configuring these things in two different places, as they should be, since runtime config should be part of the code IMO.
It would make a lot more sense and would be more explicit IMO to use System.get_env in your code than in a config file.
This is the most recent thing we tried in Phoenix and Ecto via the init/1 callback and it caused a lot of confusion because configuration is now broken in two places. So someone would try to set a static flag in their config.exs and nothing would happen. They would spend some time debugging the problem and it turned out that the flag was being overridden in the code with an environment variable.
Although one is runtime and the other is compile-time, they are still configuration, and having them in two completely different places can lead to the confusion I above.
It is not much different from having separate files for compile and runtime, which was proposed earlier. It is a valid approach and with more flexibility, but it will likely lead to file explosion too.
Indeed, it is similar, but it does solve two issues that the separate-files proposal had:
I have a politics related question: how willing is Elixirâs core team to introduce a breaking change that forces everybody to migrate to a seemingly perfect solution (which is yet to be discovered)? This should only come with the next major version however â say, Elixir 1.7.
My experience with people says that they will not change their configuration files for years and will only do so if a version upgrade mandates it. Due to that, itâs my opinion that when the solution is found it definitely should be made very explicit and impossible to ignore. Otherwise a lot of library authors will keep coming up with creatively flawed solutions forever.
Whatâs your opinion on this political matter?
EDIT: Obviously, a transitional period with deprecation warnings should be around for a while before a breaking change is introduced.
Iâve had such issues myself with just config scripts. In particular I went down a couple of wild goose chases because I didnât notice an override in another .exs script, or there was an override in the code (either as a compile-time if on Mix.env, or as a runtime if on some other non-obvious global).
Therefore, the problem is IMO not in init/2 but in the fact that parameters for endpoint are dispersed across multiple places (which they already were even before init/2 was introduced). If on top of it people used one more place for providing parameters, Iâm not surprised they were even more confused.
Iâd say though that this wasnât a proper evaluation of init/2. Iâd wonder if the people would face the same problems if they provided the complete parameters to the endpoint in a single place (init/2 callback).
If the guidelines for library authors say we should not use application configuration unless strictly necessary, then we should make it less-convenient for them to do that, so that we have less problems with configuration being in multiple places. It feels like the proposal being discussed in this thread makes it easier for both library users and library authors to continue to not follow that advice.
Additionally, introducing a dynamic step that happens before application boot to run your configuration script will introduce a new failure mode that would presumably be outside of the normal supervision hierarchy. This leaves the developer with less opportunity to deal with failures and less opportunity to bring up parts of the system that they want to be online even if the configuration is invalid or some âside-effectâ of the configuration script fails on first run, using the normal OTP toolbox that we have today.
© Copyright Elixir Forum | Terms | Privacy & Cookies
