Proposal: moving towards discoverable config files

[bitwalker]

@josevalim

I have a hunch that we are talking past each other here.

Haha probably

I guess @OvermindDL1 came closest to my feelings on compile-time configuration, which is that config.exs feels like the wrong place for that kind of thing. People have had a tendency to use config.exs for all configuration, compile-time (i.e. baked into the build), runtime global, and runtime local (i.e. really should be a parameter to a supervisor spec, or a function, but is instead read from the app env).

Why not use mix.exs for compile-time configuration? It is already used for a variety of compile-time only config items, such as :elixirc_paths, :compilers, etc. Some of that blends with project configuration, but as you can see from one of the examples you provided, it is not uncommon to see references to Mix.Project.config in config.exs, which implies that project configuration can be seen as just another set of compile-time config values. Not to mention that you can already do “compile-time” configuration for your own application in mix.exs:

def application do
  [env: [key: value, ...]]
end

I guess my beef is with the fact that we are looking to make the distinction between compile-time and runtime configuration in config.exs with on_boot; and I think we should re-evaluate why we think config.exs is the right place for compile-time config in the first place. @OvermindDL1 has some interesting thoughts on that, and I think there is something to be said for how Go, Java, and others deal with it. You can set application env variables via the command line already, so I think we have that covered, but what we can’t do is define compile-time constants, for example the current commit hash as demonstrated in one of the examples earlier. I think virtually all the examples we’ve seen here fall in that category. Having a convenient place to set those, or set defaults, seems like a sane choice, and what better place than mix.exs, which effectively already carries some of those constants (app name, version, etc.).

If we are going to view config.exs as compile-time config though, I think that is confusing, and is going to continue to be confusing, even with additions like on_boot. I’ve always viewed config.exs in the same way that I view .conf, .ini and other config files in that vein - and I suspect I’m far from alone in that.

[OvermindDL1]

@bitwalker

If we are going to view config.exs as compile-time config though, I think that is confusing, and is going to continue to be confusing, even with additions like on_boot . I’ve always viewed config.exs in the same way that I view .conf , .ini and other config files in that vein - and I suspect I’m far from alone in that.

As an aside, this surprised me too. I quickly realized ‘how’ it works, but it still seems super weird to me to this day and I still mentally try to parse it as a run-time configuration instead of a build-time configuration…

[josevalim]

@bitwalker

Why not use mix.exs for compile-time configuration?

I really don’t think that the choice of my Ecto adapter or my logger compile_purge_level belong in mix.exs. The mix.exs is about how Mix compiles my project and Mix as a build tool in general, it is not about how my application interfaces with something as important as a logger.

I also don’t believe that users should have to know or care if a configuration is compile time or runtime unless they really have to. Imagine how much confusion we will have if now my Logger configuration is spread out in two places. The only time not knowing if a configuration happens at runtime or compile-time will be an issue is if the user expects something to happen at runtime but that is a compile time only option. That’s exactly where on_boot can be helpful: if we allow the user to declare their expectations via something like on_boot, we can detect those cases programatically.

@bitwalker

Not to mention that you can already do “compile-time” configuration for your own application in mix.exs :

No, you can’t configure logger in your own def application do.

@bitwalker

I guess my beef is with the fact that we are looking to make the distinction between compile-time and runtime configuration in config.exs with on_boot

This issue is not about on_boot at all. Supporting releases is what will make the distinction between compile-time and runtime exist. The current proposal, as originally written, has the exact same compile vs. runtime issue as on_boot, except it is implicit and left up for the users to guess. on_boot is about making those differences explicit.

@bitwalker

I think we should re-evaluate why we think config.exs is the right place for compile-time config in the first place.

I will be straight-forward here: this is out of discussion. config.exs is executed at compilation time. It has been behaving like this for at least 4 years. It doesn’t matter how people think it behaves. Compile-time is how it behaves right now and we are not trying to change it. And libraries like Nerves, Phoenix, Ecto and Logger rely on this behaviour at different places. Suggesting to find a new home for all of those common configuration patterns would require a long-term migration effort (as both libraries and config files need to be rewritten). It is impractical at best (and IMO just plain undesirable). While I would agree some of this compile-time configuration should actually be moved to lib (such as the Ecto adapter), those few cases are rather the exception.

In fact, all of the alternatives brought up so far, such as command line application configuration, is not related to compile-time configuration, but rather to runtime configuration. In a release, the only thing I can specify at the command line, by definition, is runtime configuration. So if we really don’t want to mix compile time vs runtime, then the only option is to keep the config.exs as is and outside of releases, and use tools like command line options, Phoenix’ init/2 callback and friends for the runtime configuration.

[talentdeficit]

In fact, all of the alternatives brought up so far, such as command line application configuration, is not related to compile-time configuration, but rather to runtime configuration. In a release, the only thing I can specify at the command line, by definition, is runtime configuration. So if we really don’t want to mix compile time vs runtime, then the only option is to keep the config.exs as is and outside of releases, and use tools like command line options, Phoenix’ init/2 callback and friends for the runtime configuration.

i like this proposal best of all. retain config.exs for compile time only and let it continue to work as it does. move runtime config to some other place

i think the confusion users experience in the difference between compile time and runtime is entirely a result of encouraging the use of the same file/process for both. separating them is probably the best thing that can be done to resolve this

[josevalim]

@blatyo

I assumed the import_config change was a required step for supporting releases. If just on_boot were done, wouldn’t a release still not know where to pull in config?

The import_config change is necessary if we want to execute the same files in a release as we do for Mix. So we need to track the files. If we have on_boot, we only need to track the code inside on_boot.

@talentdeficit

i think the confusion users experience in the difference between compile time and runtime is entirely a result of encouraging the use of the same file/process for both. separating them is probably the best thing that can be done to resolve this

The reason the confusion exists is because the compile/runtime is completely irrelevant for Mix but it exists in a release. Doing nothing means the major mismatch we see between Mix and releases in regards to config won’t be addressed.

Unless we strictly impose Mix to be a compile-time config only. This means that MIX_ENV=prod PORT=4040 mix run won’t have any effect if the project was already compiled with PORT=8080. But going down this path feels like we are breaking both approaches, instead of fixing them.

[OvermindDL1]

@josevalim

In fact, all of the alternatives brought up so far, such as command line application configuration, is not related to compile-time configuration, but rather to runtime configuration.

Eh, only in so far as mix is ‘running’ like ELIXIR_BLAH='something' mix compile or mix compile -DELIXIR_BLAH=something or so.

@josevalim

The reason the confusion exists is because the compile/runtime is completely irrelevant for Mix but it exists in a release.

Actually the definition is intensely important in normal code as well. Like the URL for the ueberauth_cas library gets hardcoded into the compiled code at compile time from the config, when it by all accounts should be runtime changeable (and I had to have it to that, hence forked). Yes this is a library issue, but the current style ‘encourages’ this usage, which is a big problem.

@josevalim

Unless we strictly impose Mix to be a compile-time config only. This means that MIX_ENV=prod PORT=4040 mix run won’t have any effect if the project was already compiled with PORT=8080 .

Except an environment variable should take precedence over a higher staged default value, thus 4040 should be used over the 8080 unless the code does something highly irritating like bake it into the compiled beam (which, again, is what the current config system encourages people to do)…

[alchemist_ubi]

I am not a big fan of adding any complexity to the mix.exs file I would prefer to go with the on_boot macro and keep all the config related code in the config files.

I am kind of worry about library creators adding the configuration to their mix.exs file.

How that would work then?

Right now it is simple and I dont worry about config files in my deps because they are not being used.

Are you gonna ignore that config as well?

That is my main concern when you push anything to the mix.exs file instead of living it in the config files.

But again going back to the first sentences, no mix.exs has metadata that shouldn’t be important for must of the library packages and I would love to keep it that way.

And is better to rely on copy-paste configs than library authors handling all possible use cases for the configs this is my biggest concern.

@bitwalker

Configuration should be mostly static in nature

Definitely what I expect from configuration perspective, but

config_path or config_files because it seems that every config is actually a file, no a folder so that confuse me from language perspective. Sorry for the dumb question.

@hubertlepicki

For the simple umbrellas many people would be happy with just top-level config files

Just a comment: Only for production release, I wouldn’t like to have my test configs per app in the root of the umbrella app (it becomes too large with small benefits)

[alchemist_ubi]

Standalone preferences so far,

Add macros to the config files like on_boot.

DevOps and engineers have one place to deal with configuration issues and library authors do not have to deal with configurations for you specific use case (unless you ignore every config from the mix.exs proposed so far)

[josevalim]

@OvermindDL1

Yes this is a library issue, but the current style ‘encourages’ this usage, which is a big problem.

Exactly. The only time a user cares about runtime/compile is when they want to configure something compile-time at runtime. on_boot would make those cases clearer, hence this discussion. If in the future we add metadata to applications that allow them to tell which configuration happens at compile-time, we would even be able to catch those errors programatically early on.

In any case, it is likely that the configuration you mentioned shouldn’t even be in the application environment at all, regardless if it is compile-time or runtime. Because even if you move something to be a runtime configuration, it is still a global setting, that we want to avoid nonetheless. Hence the latest recommendations in the library guidelines.

At the end of the day, the fact there are some illegitimate cases of compile-time configuration does not imply that they aren’t legitimate cases for compile-time configuration. Those are quite common and we are not breaking them. If we want to make it harder to leverage compile-time configuration, then we need to make it harder for library authors, not users. We shouldn’t “punish” users because of mistakes library authors make.

[cybrox]

@OvermindDL1

Currently, the {:system ,value} tuple exists as a de-facto standard in many situations.

No, no it’s not, it would be nice if it was, and sure the usual phoenix/ecto and so forth libraries support it, but so so very few third-parties support it.

You’re definitely right about that, I agree that it is not used consistently enough. The point I was trying to make is, that it is an already existing, elegant and simple possibility of solving a large part of the issue at hand. - Albeit not as clean as solutions proposed here.

@OvermindDL1

However, it is usually only used in the production environment.

I don’t see why? I use the system environment excessively for production and development (and rarely, but occasional on test too).

I assume this is mostly because a lot of people try to stick to paradigms set by the “big players” in terms of frameworks / libraries and follow the phoenix style of splitting configs per-env, which allows for hard-coding those parameters for development environments. Personally, I agree with you, I use environment variables for configuration in development as well.

@OvermindDL1

I could do ELIXIR_CONFIG=prod.json mix phx.server or so to specify some things to override the internal config defaults…

This is actually a very interesting idea, imo. It would allow for extending the current possibilities in a much more dynamic way without adding bloat and complication to the mix.exs file.

---

@bitwalker

People have had a tendency to use config.exs for all configuration, compile-time (i.e. baked into the build), runtime global, and runtime local (i.e. really should be a parameter to a supervisor spec, or a function, but is instead read from the app env).

This is probably the most important point of what sparked the need for a proposal like this. I can’t help but think that introducing a parallel mechanism for doing things (Application.config) will add more complexity than required. The current set of tools would already suffice for solving this issue, if only we would come to an agreement on how to split those entirely different parts of configuration.

@josevalim

I also don’t believe that users should have to know or care if a configuration is compile time or runtime unless they really have to.

I would like to disagree, depending on how libraries are implemented, this difference is crucial to ones understanding of their own configuration. I that regard, I personally think the current way of config option x is compile-time, unless explicitly implemented otherwise is very clear. The big issue is, that most people are not aware how - and a lot of libraries don’t allow - to even use runtime configuration.

@josevalim

I will be straight-forward here: this is out of discussion. config.exs is executed at compilation time.

@josevalim

Unless we strictly impose Mix to be a compile-time config only. This means that MIX_ENV=prod PORT=4040 mix run won’t have any effect if the project was already compiled with PORT=8080 . But going down this path feels like we are breaking both approaches, instead of fixing them.

Yep. As someone who has working tooling for my applications in place, a change like this would be terrible. We are trying to solve an issue that is not ideal, it’s not something that’s broken, thus adding possibilities instead of just replacing working solutions would definitely be the way to go.

@josevalim

Exactly. The only time a user cares about runtime/compile is when they want to configure something compile-time at runtime.

Exactly. I think @OvermindDL1’s proposal for dynamically importing one other configuration file would be an awesome solution for adding this possibility without introducing unnecessary complexity.

• The user can still use config/{config|env}.exs for compile-time config
• The user can still use System.get_env/1 etc. there, not breaking existing configs
• The user is more aware of the static nature of those configurations
• The user can explicitly pick a config without additional code

tl;dr: I really like RUNTIME_CONFIG=config/myconfig.exs bin/myapp start

[sasajuric]

@josevalim

At the end of the day, the fact there are some illegitimate cases of compile-time configuration does not imply that they aren’t legitimate cases for compile-time configuration.

Personally I believe that a lot of the stuff which is managed by config.exs & friends belongs to the code. For example these are params which mix phx.new by default pushes into config script, spread across four files, bundled with unrelated data, but detached from the place where it logically belongs.

Why should by default this data reside in config scripts, and why at runtime must it sit in a public global mutable storage in a separate memory space?

@josevalim

We shouldn’t “punish” users because of mistakes library authors make.

No, but we could guide users to use approaches which make sense. I think that a lot of misuse of config scripts originates from the fact that generators promote them by default.

[stevedomin]

@josevalim

Unless we strictly impose Mix to be a compile-time config only. This means that MIX_ENV=prod PORT=4040 mix run won’t have any effect if the project was already compiled with PORT=8080 . But going down this path feels like we are breaking both approaches, instead of fixing them.

@josevalim Is that feasible? Apologies if this is out of topic but I feel like a lot of the confusion comes from Mix and how it’s used, or at least how we started using it as a community.

Given that Mix is a build tool it seems logical to me that the configuration living in config.exs should be compile-time only. It should only contain the configuration useful to build my project (which pubsub adapter I want to use, etc.).

I’d argue that it’s the same for Mix tasks. In his “Elixir Deployment tools update”, @bitwalker talks about adding support for Mix tasks, and while I like the idea I kinda feel like maybe we shouldn’t be using Mix tasks for what we are using them for (running migrations, etc.)

@sasajuric

Personally I believe that a lot of the stuff which is managed by config.exs & friends belongs to the code. For example these are params which mix phx.new by default pushes into config script, spread across four files, bundled with unrelated data, but detached from the place where it logically belongs.

Completely agree with that, although I do think having a centralised place for operator configuration might be desirable. Or at least a centralised place to discover what’s configurable. As an operator I shouldn’t have to look at hundreds of file to know what I can configure and how.

[sasajuric]

@stevedomin

I do think having a centralised place for operator configuration might be desirable. Or at least a centralised place to discover what’s configurable. As an operator I shouldn’t have to look at hundreds of file to know what I can configure and how.

I agree. If you want to assist operation, then it needs to be done. However, this will require an explicit effort to pick the things which are relevant to the operator and extract them into a separate place. Most of the stuff currently sitting in config scripts is plain noise for the operator, while some of the stuff they need might likely not be there.

I completely agree with @michalmuskala here:

@michalmuskala

Things that are usually operator configuration - API keys, component addresses should not go in there.

[josevalim]

@cybrox

tl;dr : I really like RUNTIME_CONFIG=config/myconfig.exs bin/myapp start

The problem with this approach, as far as I understand it, is that it does not solve the issue originally written in this proposal: which is the gap between Mix and releases. What you are proposing is that, once you assemble a release, you need to explicitly list all of the runtime configurations in a separate file, still causing confusion on why my release does not work and leading to duplication. For example, per your proposal above, imagine that I want to do this in a Phoenix app:

config :my_app, Endpoint, port: System.get_env("PORT") || 9000

How would I make it work on both Mix and in a release? Does it mean that I have the line above in my config/prod.exs (which is compile time and Mix based) and then have the same line in a separate runtime config for releases?

@sasajuric

No, but we could guide users to use approaches which make sense. I think that a lot of misuse of config scripts originates from the fact that generators promote them by default.

The config scripts are not the root cause. What is being misused is the application environment. The fault is in library authors using application environment or forcing compile time configurations when they don’t have to. If library authors don’t abuse the application environment, then the configuration issue is immediately solved because there will be nothing to configure. That’s why I am saying I would rather push the complexity to library authors. For users of libraries, configuration should be as simple as it gets.

@sasajuric

Why should by default this data reside in config scripts, and why at runtime must it sit in a public global mutable storage in a separate memory space?

It doesn’t matter for this discussion. We had this discussion at the beginning of the thread. First, not everyone agrees with this line of thought (I can elaborate in a separate thread as I don’t think it belongs here). The second issue is that, regardless of how many libraries promote wrong usage of the application environment, we will still have valid use cases. We could talk about Phoenix all we want, but libraries like Logger and the MIME library still correctly use compile-time configuration, and for those libraries configuration should be as first class as possible.

[sasajuric]

@josevalim

The config scripts are not the root cause. What is being misused is the application environment.

But isn’t today the sole (or at least the main) purpose of config scripts to populate app env? What else are they used for?

@josevalim

First, not everyone agrees with this line of thought

It was not a line of thought, it’s an honest question. I seriously don’t get why should this be a recommended default?

@josevalim

I can elaborate in a separate thread as I don’t think it belongs here

Yes please, I already opened a separate thread yesterday.

@josevalim

libraries like Logger and the MIME library still correctly use compile-time configuration, and for those libraries configuration should be as first class as possible

I agree here. My impression is though that such situations are not all that frequent, and that config scripts are dominated by things which shouldn’t be there. I don’t have a significant sample though

[cybrox]

@josevalim

Does it mean that I have the line above in my config/prod.exs (which is compile time and Mix based) and then have the same line in a separate runtime config for releases?

No, I agree that duplication should be avoided and after thinking that approach through again, I can see the issues you pointed out. Especially the environment-dependent configuration in the config/prod.exs would cause a collision if it was just re-defined in an external config file.

After reading through the original proposal again, with the things discussed later on in mind, the idea is starting to grow on me. However, I still can’t help but think that it will become incredibly confusing for the user, especially if it is not implemented across all libraries consistently. Sensitise library authors for consistent usage is probably the single most important aspect to this proposal’s success.

Also, as @hubertlepicki pointed out, I think the biggest challenge would be defining which values have to be changed on boot time. There are lots of obvious ones for sure, but also some gray areas.

Would Application.Config allow for incremental configuration? I very much like the idea of config/config.exs becoming a reference point for configurable options by setting all the required values and configurations loaded afterwards can overwrite these.

[josevalim]

@sasajuric

But isn’t today the sole (or at least the main) purpose of config scripts to populate app env? What else are they used for?

We only have to populate the app env because libraries read things from the app env. If libraries do not read from the app env, we won’t have anything to populate. Who is reading from the app env is the root issue, not who is writing to it.

Writing configuration (config.exs) should always be straight-forward. We can talk about ways to make library authors read from the env less frequently but that is a separate discussion.

@cybrox

Also, as @hubertlepicki pointed out, I think the biggest challenge would be defining which values have to be changed on boot time. There are lots of obvious ones for sure, but also some gray areas.

Our goal with on_boot is to capture the user intent. It means the user would like those to be set at runtime. The user shouldn’t have to care what happens at compile time and what happens at runtime from the library side of things unless the user intent conflicts with the library behaviour. Ideally we will find ways in the future of checking the user intent with the actual implementation programatically.

@cybrox

Would Application.Config allow for incremental configuration? I very much like the idea of config/config.exs becoming a reference point for configurable options by setting all the required values and configurations loaded afterwards can overwrite these.

With we go with on_boot, that will continue to work the same as today. If we go with the original proposal, then we will need to slightly flip things around as outlined in the proposal.

[Qqwy]

I finally was able to find the time to read through this whole thread. The one thing everyone agrees on, is that the current way configuration happens could be improved. The exact opinions people then have about how to improve it and what to improve on exactly are then widely varying; I guess we might still for some of the proposed issues only be looking at a symptom rather than the underlying cause.

Of course, the proposal at hand is only about one single thing (there not being a real way to do currently writing runtime configuration for releases), so let me focus on that one now in more detail. The other issues that have been discussed so far (what to (not) put in a configuration file, other ways to configure an application, how applications ought to read configuration settings that the user put in, and some others) are I think very important, but definitely fall outside of this thread’s intended subject matter.

---

It is an unmistakable fact that an Elixir project is first compiled, and then run. In some cases, the environment that the application is compiled on is different from the environment where the application is run. This is true for at least releases and Nerves projects. In all cases (not only in these two), both of these steps (compilation and booting+running) happen. Even when we run a project in a Mix project, it is first compiled and only then executed. However, since we usually happen to still be in the same folder with the same files (+environment) available, we can accidentally (e.g. without being conscious about it) depend in our run-time on things we decided during compile-time: re-use our configuration.

So currently, Mix configuration happens in both places in Mix projects because it works ‘by happy circumstance’ also during application startup, but it does not in the other cases (actually, I think this is regardless of the fact of having Mix still available to us at runtime or not, because we cannot depend on the locations of files or other things anymore when the runtime environment(/machine) is different from the compilation environment(/machine)).

To me, it thus feels like the current behaviour is implicit, and it can be made more explicit by making it clear that the difference between compile-time configuration and run-time configuration exists (and has always existed!) and needs to be kept in mind.

The two proposals we currently have, Application.Config and on_boot both only address this problem partially:

• Application.Config solves the problem of Mix not always being available, but hides the fact that compile-time and run-time configuration might be different (or, maybe clearer phrasing: that some beam-applications might read parts of the configuration during compile-time and some others during boot-up or runtime).

• on_boot makes it more clear that certain pieces of configuration will only be executed during boot-up/runtime, but to be honest it does not feel very clean to me because of the following fact: It seems to indicate that something different needs to happen during boot-up/runtime rather than the compile-time configuration outside of it. it is the word different that I have issue with here.

It feels to me that the ‘default’ of an application would be to read the configuration settings during boot-up/runtime, and only if the application requires special behaviour to happen during compilation-time (because of time- or space-optimizations), then this is where we would opt-in into a special compile-time configuration block, rather than compile-time being the default and opting-in into a special run-time configuration block.

It is only a thought however, because I do see that with the way how configurations currently work (Releases can act on config during compile-time but not during boot-up/runtime), that it might be difficult to build it with this order of specialization (compile-time config being a specialization of run-time config) in mind. But I do think that it is important to mention this, because personally, on_compile feels clearer to me than on_boot.

---

But besides this, in essence I think the problem is not that ‘releases are broken’ but rather that Mix is able to pretend that the difference between compile-time and run-time configuration does not exist. So rather than building a ‘fix’ for releases, I think that improving on Mix’ behaviour to make it more clear that there is (and always has been) a difference is the way to go.

[arturcygan]

At first I was quite enthusiastic about bringing mix config and releases closer together. I’ve struggled quite hard when we started with releases and getting nils on production from the System.get_env/1 in prod.exs was very confusing. But now after giving it some thought, I’m no longer sure if we actually need to close the gap between mix project and a release. A release is a fundamentally different thing, it is not a project on a development machine, it is a build artifact with different structure, ready to be deployed on a production server. Whether the development team or a separate ops team operate the release I believe loading the prod mix config on the server provides a little value for them. Mix is a build tool and configs in config/ directory are build configs. So I agree with Michał here:

@michalmuskala

I don’t agree with the premise that mix config is for operator configuration - it is most definitely not. Things that are usually operator configuration - API keys, component addresses should not go in there.

Instead of trying to make mix project and release transparent maybe we should embrace and document their differences? Better education on the topic, release tooling and defaults which guide a newcomer. Take the config/prod.secret.exs for example. It is very strong suggestion that we should deploy a source code to the server, not a release, because only mix can load this config file. Making the build config a hybrid with a runtime config will maybe make the initial deploys a bit easier but in a longer run it might be even more confusing than the current situation.

I believe we need to treat runtime config seriously and explicitly, without ad-hoc solutions. If you look at most of services (like postgres, nginx etc.) we use, they provide strict configuration through external measures (file, env vars) which has well defined interface. Are the applications we build different?

What about the following solution? There is a sys.config which could be treated as static “readonly” config, a transpiled prod.exs with “safe defaults”. Loading configuration might grow to complex strategies as mentioned by OvermindDL1 and needs dedicated flexible approach with different backends forming fallback stack. Backends should be coded and tested as regular code, as they are general system ↔ operator (this can be even us during development) contract. In dev.exs and test.exs we can disable all the backends because we can simply put config “straight” from the mix config and enable in prod.exs config (this is a compile time decision). This would require injection in early stages of execution to put config from backends before the system starts (similarly as mix does). In case of enabling config backends in mix project, mix config provides a base which will be overrided by config backends, while running a release the base would be sys.config.

This is an awesome community discussion!

[ConnorRigby]

I’m the primary author of the FarmbotOS Nerves project. I’m still reading thru this entire thread, but i also agree that Elixir has issues with configuration. I had always that was inherited from Erlang, since it suffers many of the same problems.

The following are my opinions on what I’ve read so far as an end user of the Mix/config system in varying use cases. I apologize in advance if it is a little jargony feel free to ask and i will try to clarify anything. I’ve separated it this way to make a point. Right now the current Mix.Config system is applicable to all of these very different deployment strategies, and being so I do not believe there will be a system that 100% envelops all of them.

Building a Nerves based project

This is one of the situations i spend most of my time, and it may be a bit long winded.
As the original post described I’ve had to do some weird things as to suit application requirements. One example of this is retrieving things
in Mix.Project.config() such as:

• application version
• application environment (dev, prod, test etc)
• nerves target (host, rpi3, my_custom_hardware, etc)

Now since those are in fact compile time things, it might look weird that i have a separate module tracking their values, this comes back to a shortcoming in Nerves currently - we don’t have a code reloader yet, so during development, i frequently do something a little scary that i wont go into detail here. The gist of it is that if i recompile a module while working on a nerves device, Mix is not available, so those fields are also not available.

The other thing that i would like to point out is that in my Nerves based projects i often find the need to not only retrieve config at runtime but also set it. For the most part i use something like Ecto/Sqlite for this, but i do have a bad habit of storing data globally in Application.put_env and getting it with Application.get_env. A tangible example of this is interfacing with hardware. Elixir lends itself to behaviours for hardware nicely. In the Farmbot project we interface with a UART device. it’s not always possible to have the device plugged in during development so doing something along the lines of Application.get_env(:farmbot, :firmware_handler) works really well. I can listen for the device being plugged in, replace that value with the Firmware.UARTHandler module and restart the firmware stack. When it’s unplugged, replace the value with Firmware.StubHandler. You get the idea. Now is reconfiguration at runtime something that Mix/this new thing needs to handle or even be aware of? i don’t know to be honest, but this does feel hacky to me anyway.

Those are the two main experiences i have with configuration of Nerves projects.

Building a library specifically targeting Nerves projects

This section encapsulates projects such as nerves_network, nerves_init_gadget and the like. This is usually where i experience the most pain and it is not particularly the fault of Mix.Config, but it does allude to a big picture issue i have with Elixir/Erlang libraries. There are to many ways to configure an application, and end users of the library often expect to be able to configure how they see fit. Let’s take nerves_network as an example.

you can configure Nerves network in two different ways.

use Mix.Config

config :nerves_network, :default, [
   eth0: settings
]

or you can do

iex()> Nerves.Network.setup(:eth0, settings)

This is a nightmare handle as a library developer.

Building a general use Elixir library

The Nerves specific issue i denoted above applies here, but I’ve found it is less of an issue for generic libs. One issue i have with this use case is that lib developers frequently use sub dependencies. If those sub dependencies need configuration, it is forced upon the end user of your library to configure them.

Building a generic Elixir application

To me this is where the current system shines in my opinion. I don’t have many complaints with the caveat that i don’t usually “deploy” these applications. (so no distillery, or similar).

Building a Phoenix based project

I don’t have the most experience with Phoenix, but I do work on two production apps. One deployed on Heroku, one deployed on Gigalixir. Both share a common issue of using environment variables to replace the things in Mix.Config.

Closing thoughts

After rereading this post it seems like i’m just crying and i don’t really have any offered solutions or ideas. Truth be told i don’t actually know what my ideal api would be for each of the listed use cases and i don’t actually believe there is one all encompassing api that would handle all of them 100%. There is certainly going to be trade offs and balances to be struck. I’m afraid that changing the current system will only fragment the issue rather than push toward a good solution for everyone.