Proposal: moving towards discoverable config files

You’re definitely right about that, I agree that it is not used consistently enough. The point I was trying to make is, that it is an already existing, elegant and simple possibility of solving a large part of the issue at hand. - Albeit not as clean as solutions proposed here.

I assume this is mostly because a lot of people try to stick to paradigms set by the “big players” in terms of frameworks / libraries and follow the phoenix style of splitting configs per-env, which allows for hard-coding those parameters for development environments. Personally, I agree with you, I use environment variables for configuration in development as well.

This is actually a very interesting idea, imo. It would allow for extending the current possibilities in a much more dynamic way without adding bloat and complication to the mix.exs file.


This is probably the most important point of what sparked the need for a proposal like this. I can’t help but think that introducing a parallel mechanism for doing things (Application.config) will add more complexity than required. The current set of tools would already suffice for solving this issue, if only we would come to an agreement on how to split those entirely different parts of configuration.

I would like to disagree, depending on how libraries are implemented, this difference is crucial to ones understanding of their own configuration. I that regard, I personally think the current way of config option x is compile-time, unless explicitly implemented otherwise is very clear. The big issue is, that most people are not aware how - and a lot of libraries don’t allow - to even use runtime configuration.

Yep. As someone who has working tooling for my applications in place, a change like this would be terrible. We are trying to solve an issue that is not ideal, it’s not something that’s broken, thus adding possibilities instead of just replacing working solutions would definitely be the way to go.

Exactly. I think @OvermindDL1’s proposal for dynamically importing one other configuration file would be an awesome solution for adding this possibility without introducing unnecessary complexity.

  • The user can still use config/{config|env}.exs for compile-time config
  • The user can still use System.get_env/1 etc. there, not breaking existing configs
  • The user is more aware of the static nature of those configurations
  • The user can explicitly pick a config without additional code

tl;dr: I really like RUNTIME_CONFIG=config/myconfig.exs bin/myapp start

2 Likes

Personally I believe that a lot of the stuff which is managed by config.exs & friends belongs to the code. For example these are params which mix phx.new by default pushes into config script, spread across four files, bundled with unrelated data, but detached from the place where it logically belongs.

Why should by default this data reside in config scripts, and why at runtime must it sit in a public global mutable storage in a separate memory space?

No, but we could guide users to use approaches which make sense. I think that a lot of misuse of config scripts originates from the fact that generators promote them by default.

5 Likes

@josevalim Is that feasible? Apologies if this is out of topic but I feel like a lot of the confusion comes from Mix and how it’s used, or at least how we started using it as a community.

Given that Mix is a build tool it seems logical to me that the configuration living in config.exs should be compile-time only. It should only contain the configuration useful to build my project (which pubsub adapter I want to use, etc.).

I’d argue that it’s the same for Mix tasks. In his “Elixir Deployment tools update”, @bitwalker talks about adding support for Mix tasks, and while I like the idea I kinda feel like maybe we shouldn’t be using Mix tasks for what we are using them for (running migrations, etc.)

Completely agree with that, although I do think having a centralised place for operator configuration might be desirable. Or at least a centralised place to discover what’s configurable. As an operator I shouldn’t have to look at hundreds of file to know what I can configure and how.

2 Likes

I agree. If you want to assist operation, then it needs to be done. However, this will require an explicit effort to pick the things which are relevant to the operator and extract them into a separate place. Most of the stuff currently sitting in config scripts is plain noise for the operator, while some of the stuff they need might likely not be there.

I completely agree with @michalmuskala here:

3 Likes

The problem with this approach, as far as I understand it, is that it does not solve the issue originally written in this proposal: which is the gap between Mix and releases. What you are proposing is that, once you assemble a release, you need to explicitly list all of the runtime configurations in a separate file, still causing confusion on why my release does not work and leading to duplication. For example, per your proposal above, imagine that I want to do this in a Phoenix app:

config :my_app, Endpoint, port: System.get_env("PORT") || 9000

How would I make it work on both Mix and in a release? Does it mean that I have the line above in my config/prod.exs (which is compile time and Mix based) and then have the same line in a separate runtime config for releases?

The config scripts are not the root cause. What is being misused is the application environment. The fault is in library authors using application environment or forcing compile time configurations when they don’t have to. If library authors don’t abuse the application environment, then the configuration issue is immediately solved because there will be nothing to configure. That’s why I am saying I would rather push the complexity to library authors. For users of libraries, configuration should be as simple as it gets.

It doesn’t matter for this discussion. We had this discussion at the beginning of the thread. First, not everyone agrees with this line of thought (I can elaborate in a separate thread as I don’t think it belongs here). The second issue is that, regardless of how many libraries promote wrong usage of the application environment, we will still have valid use cases. We could talk about Phoenix all we want, but libraries like Logger and the MIME library still correctly use compile-time configuration, and for those libraries configuration should be as first class as possible.

1 Like

But isn’t today the sole (or at least the main) purpose of config scripts to populate app env? What else are they used for?

It was not a line of thought, it’s an honest question. I seriously don’t get why should this be a recommended default?

Yes please, I already opened a separate thread yesterday.

I agree here. My impression is though that such situations are not all that frequent, and that config scripts are dominated by things which shouldn’t be there. I don’t have a significant sample though :slight_smile:

No, I agree that duplication should be avoided and after thinking that approach through again, I can see the issues you pointed out. Especially the environment-dependent configuration in the config/prod.exs would cause a collision if it was just re-defined in an external config file.

After reading through the original proposal again, with the things discussed later on in mind, the idea is starting to grow on me. However, I still can’t help but think that it will become incredibly confusing for the user, especially if it is not implemented across all libraries consistently. Sensitise library authors for consistent usage is probably the single most important aspect to this proposal’s success.

Also, as @hubertlepicki pointed out, I think the biggest challenge would be defining which values have to be changed on boot time. There are lots of obvious ones for sure, but also some gray areas.

Would Application.Config allow for incremental configuration? I very much like the idea of config/config.exs becoming a reference point for configurable options by setting all the required values and configurations loaded afterwards can overwrite these.

1 Like

We only have to populate the app env because libraries read things from the app env. If libraries do not read from the app env, we won’t have anything to populate. Who is reading from the app env is the root issue, not who is writing to it.

Writing configuration (config.exs) should always be straight-forward. We can talk about ways to make library authors read from the env less frequently but that is a separate discussion.

Our goal with on_boot is to capture the user intent. It means the user would like those to be set at runtime. The user shouldn’t have to care what happens at compile time and what happens at runtime from the library side of things unless the user intent conflicts with the library behaviour. Ideally we will find ways in the future of checking the user intent with the actual implementation programatically.

With we go with on_boot, that will continue to work the same as today. If we go with the original proposal, then we will need to slightly flip things around as outlined in the proposal.

2 Likes

I finally was able to find the time to read through this whole thread. The one thing everyone agrees on, is that the current way configuration happens could be improved. The exact opinions people then have about how to improve it and what to improve on exactly are then widely varying; I guess we might still for some of the proposed issues only be looking at a symptom rather than the underlying cause.

Of course, the proposal at hand is only about one single thing (there not being a real way to do currently writing runtime configuration for releases), so let me focus on that one now in more detail. The other issues that have been discussed so far (what to (not) put in a configuration file, other ways to configure an application, how applications ought to read configuration settings that the user put in, and some others) are I think very important, but definitely fall outside of this thread’s intended subject matter.


It is an unmistakable fact that an Elixir project is first compiled, and then run. In some cases, the environment that the application is compiled on is different from the environment where the application is run. This is true for at least releases and Nerves projects. In all cases (not only in these two), both of these steps (compilation and booting+running) happen. Even when we run a project in a Mix project, it is first compiled and only then executed. However, since we usually happen to still be in the same folder with the same files (+environment) available, we can accidentally (e.g. without being conscious about it) depend in our run-time on things we decided during compile-time: re-use our configuration.

So currently, Mix configuration happens in both places in Mix projects because it works ‘by happy circumstance’ also during application startup, but it does not in the other cases (actually, I think this is regardless of the fact of having Mix still available to us at runtime or not, because we cannot depend on the locations of files or other things anymore when the runtime environment(/machine) is different from the compilation environment(/machine)).

To me, it thus feels like the current behaviour is implicit, and it can be made more explicit by making it clear that the difference between compile-time configuration and run-time configuration exists (and has always existed!) and needs to be kept in mind.

The two proposals we currently have, Application.Config and on_boot both only address this problem partially:

  • Application.Config solves the problem of Mix not always being available, but hides the fact that compile-time and run-time configuration might be different (or, maybe clearer phrasing: that some beam-applications might read parts of the configuration during compile-time and some others during boot-up or runtime).

  • on_boot makes it more clear that certain pieces of configuration will only be executed during boot-up/runtime, but to be honest it does not feel very clean to me because of the following fact: It seems to indicate that something different needs to happen during boot-up/runtime rather than the compile-time configuration outside of it. it is the word different that I have issue with here.

It feels to me that the ‘default’ of an application would be to read the configuration settings during boot-up/runtime, and only if the application requires special behaviour to happen during compilation-time (because of time- or space-optimizations), then this is where we would opt-in into a special compile-time configuration block, rather than compile-time being the default and opting-in into a special run-time configuration block.

It is only a thought however, because I do see that with the way how configurations currently work (Releases can act on config during compile-time but not during boot-up/runtime), that it might be difficult to build it with this order of specialization (compile-time config being a specialization of run-time config) in mind. But I do think that it is important to mention this, because personally, on_compile feels clearer to me than on_boot.


But besides this, in essence I think the problem is not that ‘releases are broken’ but rather that Mix is able to pretend that the difference between compile-time and run-time configuration does not exist. So rather than building a ‘fix’ for releases, I think that improving on Mix’ behaviour to make it more clear that there is (and always has been) a difference is the way to go.

15 Likes

At first I was quite enthusiastic about bringing mix config and releases closer together. I’ve struggled quite hard when we started with releases and getting nils on production from the System.get_env/1 in prod.exs was very confusing. But now after giving it some thought, I’m no longer sure if we actually need to close the gap between mix project and a release. A release is a fundamentally different thing, it is not a project on a development machine, it is a build artifact with different structure, ready to be deployed on a production server. Whether the development team or a separate ops team operate the release I believe loading the prod mix config on the server provides a little value for them. Mix is a build tool and configs in config/ directory are build configs. So I agree with Michał here:

Instead of trying to make mix project and release transparent maybe we should embrace and document their differences? Better education on the topic, release tooling and defaults which guide a newcomer. Take the config/prod.secret.exs for example. It is very strong suggestion that we should deploy a source code to the server, not a release, because only mix can load this config file. Making the build config a hybrid with a runtime config will maybe make the initial deploys a bit easier but in a longer run it might be even more confusing than the current situation.

I believe we need to treat runtime config seriously and explicitly, without ad-hoc solutions. If you look at most of services (like postgres, nginx etc.) we use, they provide strict configuration through external measures (file, env vars) which has well defined interface. Are the applications we build different?

What about the following solution? There is a sys.config which could be treated as static “readonly” config, a transpiled prod.exs with “safe defaults”. Loading configuration might grow to complex strategies as mentioned by OvermindDL1 and needs dedicated flexible approach with different backends forming fallback stack. Backends should be coded and tested as regular code, as they are general system <-> operator (this can be even us during development) contract. In dev.exs and test.exs we can disable all the backends because we can simply put config “straight” from the mix config and enable in prod.exs config (this is a compile time decision). This would require injection in early stages of execution to put config from backends before the system starts (similarly as mix does). In case of enabling config backends in mix project, mix config provides a base which will be overrided by config backends, while running a release the base would be sys.config.

This is an awesome community discussion! :slight_smile:

6 Likes

:wave: I’m the primary author of the FarmbotOS Nerves project. I’m still reading thru this entire thread, but i also agree that Elixir has issues with configuration. I had always that was inherited from Erlang, since it suffers many of the same problems.

The following are my opinions on what I’ve read so far as an end user of the Mix/config system in varying use cases. I apologize in advance if it is a little jargony feel free to ask and i will try to clarify anything. I’ve separated it this way to make a point. Right now the current Mix.Config system is applicable to all of these very different deployment strategies, and being so I do not believe there will be a system that 100% envelops all of them.

Building a Nerves based project

This is one of the situations i spend most of my time, and it may be a bit long winded.
As the original post described I’ve had to do some weird things as to suit application requirements. One example of this is retrieving things
in Mix.Project.config() such as:

  • application version
  • application environment (dev, prod, test etc)
  • nerves target (host, rpi3, my_custom_hardware, etc)

Now since those are in fact compile time things, it might look weird that i have a separate module tracking their values, this comes back to a shortcoming in Nerves currently - we don’t have a code reloader yet, so during development, i frequently do something a little scary that i wont go into detail here. The gist of it is that if i recompile a module while working on a nerves device, Mix is not available, so those fields are also not available.

The other thing that i would like to point out is that in my Nerves based projects i often find the need to not only retrieve config at runtime but also set it. For the most part i use something like Ecto/Sqlite for this, but i do have a bad habit of storing data globally in Application.put_env and getting it with Application.get_env. A tangible example of this is interfacing with hardware. Elixir lends itself to behaviours for hardware nicely. In the Farmbot project we interface with a UART device. it’s not always possible to have the device plugged in during development so doing something along the lines of Application.get_env(:farmbot, :firmware_handler) works really well. I can listen for the device being plugged in, replace that value with the Firmware.UARTHandler module and restart the firmware stack. When it’s unplugged, replace the value with Firmware.StubHandler. You get the idea. Now is reconfiguration at runtime something that Mix/this new thing needs to handle or even be aware of? i don’t know to be honest, but this does feel hacky to me anyway.

Those are the two main experiences i have with configuration of Nerves projects.

Building a library specifically targeting Nerves projects

This section encapsulates projects such as nerves_network, nerves_init_gadget and the like. This is usually where i experience the most pain and it is not particularly the fault of Mix.Config, but it does allude to a big picture issue i have with Elixir/Erlang libraries. There are to many ways to configure an application, and end users of the library often expect to be able to configure how they see fit. Let’s take nerves_network as an example.

you can configure Nerves network in two different ways.

use Mix.Config

config :nerves_network, :default, [
   eth0: settings
]

or you can do

iex()> Nerves.Network.setup(:eth0, settings)

This is a nightmare handle as a library developer.

Building a general use Elixir library

The Nerves specific issue i denoted above applies here, but I’ve found it is less of an issue for generic libs. One issue i have with this use case is that lib developers frequently use sub dependencies. If those sub dependencies need configuration, it is forced upon the end user of your library to configure them.

Building a generic Elixir application

To me this is where the current system shines in my opinion. I don’t have many complaints with the caveat that i don’t usually “deploy” these applications. (so no distillery, or similar).

Building a Phoenix based project

I don’t have the most experience with Phoenix, but I do work on two production apps. One deployed on Heroku, one deployed on Gigalixir. Both share a common issue of using environment variables to replace the things in Mix.Config.

Closing thoughts

After rereading this post it seems like i’m just crying and i don’t really have any offered solutions or ideas. Truth be told i don’t actually know what my ideal api would be for each of the listed use cases and i don’t actually believe there is one all encompassing api that would handle all of them 100%. There is certainly going to be trade offs and balances to be struck. I’m afraid that changing the current system will only fragment the issue rather than push toward a good solution for everyone.

6 Likes

This is totally fine. :slight_smile:

Yes! Libraries should only require application configuration if strictly necessary. Our recent library guideline discusses this: https://hexdocs.pm/elixir/master/library-guidelines.html

If the user of your library wants to use the app environment, that should be their choice. Library authors shouldn’t impose them.

5 Likes

After reading through this thread, I’m left wondering what specific problem this proposal is trying to solve, even after it’s been summarized and re-stated several times.

  1. Introducing Application.Config that would work similarly to Mix.Config, but be present within a release where Mix isn’t available.

    I haven’t seen much discussion about this point, which perhaps indicates that people would be fine with it, assuming that it all behaves equivalently to the way it does today.

    I personally think it would be an improvement if we didn’t have config files being included dynamically, because it tends to make it difficult to figure out what the final value will be, based on several files including each other. It seems that this is more likely to be the case for Nerves-based projects than for others because we often like to work on pieces of the system on our laptops, only using Nerves to deploy the firmware to a real device at the end. That can lead to configuration file trees that pull in different files depending on whether you’re running the whole system on the device (using a release) or just running the Phoenix-based UI on your laptop (using Mix). Under the original proposal, I think it would just be a matter of putting something like this in mix.exs for eye_fw:

    # Note: @target is already set normally in a Nerves-based mix.exs file
    
    config_paths: [
      "../../eye_ui/config/base.exs",
      "../../eye_ui/config/prod.exs",
      "config/config.exs",
      "config/#{@target}"
    ]
    

    This seems fine to me. It keeps things pretty explicit as far as the order in which configurations will get applied, and allows me to not have to copy/paste the contents of the config files depending on which thing I’m trying to run.

  2. Assuming that the configuration doesn’t rely on Mix, should we apply the same config file both at compile time and runtime. I don’t recall it being specifically stated, but my understanding is that this would be a new step as a release is starting up that would process and execute the configuration somehow before starting any of the Applications in the release, so that they can have their configurations readable using Application.get_env/2 exactly as they do today, but based on the run-time environment of the release.

    There are a lot of opinions in the thread, and it’s unclear to me whether we’re specifically trying to:

    • Reduce confusion that people often have initially about whether the config.exs is applied at compile time, runtime, or both.

    • Patch over existing libraries that are doing the “wrong thing,” or allowing/encouraging their users to do the “wrong thing.”

    • Solve the question of how library authors should allow/encourage their users to specify compile-time vs run-time options and settings going forward.

      For example, would this problem be solved without making any changes at all to Mix and releases if libraries consistently supported a way for the user to specify that they want to read from an environment variable at runtime (like {:system, "DB_URL"} or just "${DB_URL}") instead of Mix literally calling System.get_env("DB_URL") in the configuration script at compile time? Could we solve the problem by doing what REPLACE_OS_VARS does, only from inside the release instead of the way it works now by rewriting the release config?

    • Solve the question of how library users should specify settings going forward. In particular, for the common case that they want to read environment variables at run-time and not at compile-time.

    • Handle all the unusual cases people currently have in their config.exs just because it’s there and it’s an executable Elixir script.

  3. Assuming that we are going to run the same configuration at compile-time and run-time, should we have a way for the user to only run certain parts of it (via on_boot) or not run certain parts of it (via on_compile).

    If the current guidance for library authors is that they should avoid using application configuration, maybe we end up needing config.exs less and less because the docs for the libraries will tell them to pass options into their init callback instead of putting something in config.exs. If that’s the future, then maybe there’s nothing to actually change about the way it works other than libraries, documentation, and community culture.

    For example, if the config.exs were only used to declare which environment variable to use and what its default value should be if not set, then it doesn’t matter whether you’re running in a release or not because it’s only declaring the name of the environment variable at both compile time and run time, not its contents. It would be up to the library to substitute the value at the time it needs to.

    What I’m getting at here is that maybe we don’t need a generic on_boot macro that can execute arbitrary Elixir code, if the real problem we’re trying to solve is how to declare that we want to read from an environment variable and what the default value should be if it’s not set.

2 Likes

It feels like we’re dealing with a leaky abstraction here.

In some cases configuration will change the code that gets compiled. e.g. A different set of macros will be run and the code generated by a compile-time configuration will have caused that. At other times, the configuration value will be used in code that doesn’t change at compile time. e.g. The host to connect to for an API. In a degenerate case, the value will be inlined as a constant at compile time and you have to go and complain to the library author that the code can’t be run in production… But that’s what the guidelines are for.

As a user I can not be expected to know which configuration will be which of the above scenarios implicitly. The fact that the MySQL Ecto adapter and the Postgrex one produce different compile-time code is “unknowable” to me because it’s based on how those libraries are implemented. You could do the same with log levels where the config changed macros and no-op’ed out everything. Clearly changing those flags at runtime is meaningless.

I guess I’m looking for the Principle of Least Surprise here. Having configuration that modified the code at compile time in a separate location from configuration that only effects runtime and enforced somehow seems like an ideal solution. (an exercise left to the reader. :smile: ) It makes the contract explicit and would require the library writer to specify the type of configuration, compile time or runtime, based on the implementation.

6 Likes

You provided a very good summary of the opinion going around the thread. To answer your bullets:

  1. Application.Config exists so we can also load configuration in a release, where Mix is not available. In a way, it is orthogonal to the other topics on this thread. Think of it as a replacement/alternative to conform.

  2. The original proposal is about applying the same config file both at compile time and runtime. This would allow us to remove both {:system, "DB_URL"} and the REPLACE_OS_VARS workarounds by literally calling System.get_env("DB_URL"). However, it doesn’t change any of the other bullet items you mentioned, in particular, it doesn’t change how library authors should read from the application environment. It may make some cases worse, for example, unusual things being done a config.exs file will now also run in a release.

  3. The on_boot is a half way step from the current proposal where, instead of running the whole config file in both compile-time and a release, a release will run only the on_boot parts. Everything else is assumed to run on compile-time, so there is no on_compile. The original proposal and on_boot aim to solve the same problem, but with different degrees of explicitness.

Excluding Application.Config discussion, we basically have four options:

  1. Run config files only at compile time (this is what happens today). This leaves users empty handed about how to configure their releases

  2. Run the same config files at both compile-time and runtime. The issue with this is that any crazy thing happening in a config.exs will now also run in production because there is no distinction what happens during compile time and what happens on boot. If you are doing something “too crazy”, it may even make booting the release fail

  3. Run config files only at compile time but allow them to declare chunks of code that should execute at runtime (on_boot). This attempts to circumvent the issues in 2 and it makes releases and Mix behave a bit closer to each other

  4. Run config files only at compile time and introduce a configuration file that is specific to releases where you would put the runtime/on_boot configuration. The issue here is that you may end-up duplicating some of the config/prod.exs in the release_config.exs

If we are talking about leaky abstraction… isn’t that forcing the leaky abstraction onto the users? :slight_smile: As a user, I would love to not have to care about this. Imagine how confusing it will be if every time I want to configure an application I need to figure which one of those buckets I need to put my configuration in. The only case where a user should have to care about this distinction is when they want to configure something that happens at compile-time during runtime.

3 Likes

Given the following:

  1. A user can set configuration data with a dynamically executed code
  2. A library might fetch the data at compile time
  3. A library might fetch the data at runtime

I don’t think you can hide the context from users. If a library needs the data at compile time, I can’t set it at runtime. And if the user is invoking e.g. System functions, then they might fail if executed in a wrong environment, or return a wrong result.

So I think something has to give here. Either configs are allowed in only one context (either runtime or compile time), or dynamic code is not supported (i.e. only constants can be provided), or users need to be aware of the context and choose where they want to provide the value. But I think that this is exactly what on_boot is solving, right?

6 Likes

Yes, exactly. There are multiple ways to solve this problem, on_boot is one of them. on_boot declares the user intent and if we allow library authors to declare what is compile-time and what is runtime, we can use this information and notify the user of any mismatches. Having separate files for runtime configs would have the same effect.

The goal with on_boot is that users do not need to know upfront what goes what. We can use the tooling to automatically detect conflicts. If I could do an analogy, forcing the users to declare what is compile vs runtime upfront would be equivalent to Git requiring me to explicitly merge all commits, even if there are no conflicts. on_boot is about bothering the user only when conflicts are detected. In order for this to work though, we need to increase the burden on library developers, which is fine and probably warranted, since most developers should rely less on compile-time application config anyway.

2 Likes

Another idea, which might make the difference between compile-time configuration vs run-time(/on-boot) configuration more clear to the user but only in the instances they have to care about it is to:

  • Introduce Application.Config which has the same behaviour as Mix.Config but lives in :elixir instead of :mix so it is available in Releases. (as in original proposal)
  • Choose which configuration files (and the order they should have) in the mix.exs file by using a :config_paths key.(as in original proposal)
  • Expose whether we want to look at the application’s mix.exs information for compile-mode or for run-time/boot-time-mode so that we can in e.g. the implementation of the mix.exs's project() fields (most importantly: :config_paths) potentially choose between different groups of configuration files. This is very similar to how different Mix environments are currently used to e.g. switch between what dependencies you include.
  • Using import_config just like you do today (including string-replacements) is still allowed, but we might decide to deprecate this feature, and definitely show warnings (or potentially hard errors) when the configuration file that was linked to can not currently be found because its string replacement was ‘too smart’, and point the user towards the new way to configure their application using a conditional config_paths: list.

I think that when we would set it up like this, the abstraction is the least leaky.
Example usage:

  1. A user that starts an Elixir project does not need to care about the difference between compile-time and run-time because exactly the same configuration is read by default (because the value for :config_paths is constant)
  2. If they start using Releases and find out that they want to perform some configuration differently at boot-time, they can change :config_paths.
  3. If they try to use configuration that is ‘too smart’ to work on boot, we can in most cases generate a warning during compilation-time (and otherwise a hard error during boot time) that should tell the user to rewrite the configuration using a conditional :config_paths setup.

In the simplest case, it would looke like:

def project do
[
  ...
  config_paths: ~w(config/config.exs config/#{Mix.env()}.exs)
  ...
]
end

Expanded to do different things based on compile-time/boot-time mode:

def project do
[
  ...
  config_paths: ~w{config/base_config.exs} ++ config_paths(Mix.env(), Mix.configuration_target())
  ...
]
end

def config_paths(_, :compile_time), do: ~w{config/compile_time_overrides.exs}
def config_paths(:dev,  :boot_time) do: ~w{config/production_boot_overrides.exs config/production_secrets.exs}
def config_paths(:prod, :boot_time) do: ~w{config/dev_boot_overrides.exs}

This proposed solution is backwards compatible, because:

  1. switching in an existing project from Mix.Config to Application.Config is optional (although we probably should make Application.Config the new default for new projects).
  2. Mix.Config will just like now work as it always did: Releases only can use it during compiletime, applications running through mix use it at compiletime + boot-time.
  3. When using Application.Config, we look at the :config_paths field in the mix.exs file. This field (and potentially other fields of the project() function) might return different results based on the current Mix.configuration_target() (Maybe we can find a better name for configuration_target?), which will allow Releases to know what files to include for boot-time, and will allow all projects to only expose the correct configuration in the correct configuration environment.
  4. Application.Config will complain if import_config is encountered inside a configuration file that is flagged for inclusion into the list of boot-time configuration. Mix.Config will do no such thing and will happily ‘just not work’ on boot-time for a Release.

Theres the “it worked on my machine” problem - someone can do something wrong but it will still work until deployed - IMO this kind of potential problems must be impossible or as edge as it gets and leaving the distinction is a step in the opposite direction.

I second that.

Users in this context are programmers, users of the language, right? Then they actually have to care unless we somehow manage to completely remove the under the hood distinction :slight_smile: on_boot solution, if adopted, already assumes that users know what it means.

That’s also my understanding but I think the users have to be aware of the context no matter what it is (if configs are only runtime or compile time only).

3 Likes

Good point. The only time they don’t have to be aware of the context is if there’s no dynamic code (i.e. only constants are provided).