It’s not something that’s purified, just some thoughts, plus I might start working sooner than I think and never get around to it, or I might do that plus improve my health and do it very soon… absolutely nothing is clear yet.
But yep, I’d like to think some more about it as well. It’s very important to clear up the value proposition before coding.
Having extracted out the initial library and having done some testing, persistent_term is clearly superior for this usecase, compared to my prior :ets solution; both in speed but also simplicity of implementation. I’m grateful to you reminding me about it early on in dev!
After some development, I’m interested in your thoughts with my prototype API. Today it looks akin to:
# config/config.exs
config :runtime, values: [
SIGNING_SALT: :string,
DATABASE_URL: :uri,
DATABASE_POOL: {:integer, min: 1, max: 100}
]
config :runtime, sources: [
json: [file: "priv/config.json"],
exs: [file: "priv/config.exs"],
dotenv: [file: "priv/config.env"],
env: []
]
While I’d love to distance myself even further from the config/*.exs world, having the library know these things in advance at compile-time lets me make very strong guarantees—for example:
Runtime.Config.get(:DATABAS_URL)Runtime.Config.get(:DATABASE_URL) will always return a %URI{} struct and contrary usage should emit a warningThis API is Keyword based, so plays well with Config’s deep-merging strategy. This means theoretically you could still do stuff like:
# config/dev.exs
config :runtime, values: [
DATABASE_URL: [
type: :uri,
required: false,
default: {:value, "http://localhost:5432/my_app_dev"}
]
]
# config/prod.exs
config :runtime, values: [
DATABASE_URL: [type: :uri, required: true]
]
or:
# config/dev.exs
config :runtime, sources: [json: [file: "priv/config/dev.json"]]
# config/prod.exs
config :runtime, sources: [json: [file: "priv/config/prod.json"]]
However, the library would generally discourage such an approach, in favor of a more robust and consolidated “context” DSL. The above examples can be written as:
# config/config.exs
config :runtime, values: [
DATABASE_URL: [
type: :uri,
required: [in: :prod],
default: [in: [dev: {:value, "http://localhost:5432/my_app_dev"}]]
]
]
config :runtime, sources: [
json: [file: "priv/config/dev.json", in: :dev],
json: [file: "priv/config/prod.json", in: :prod]
]
The current context DSL further supports things like specifying that a value is required to have been provided in contexts:
at: :compile_time | :boot_time | :runtimein: [:list, :of, :mix, :envs],for: [:list, :of, :mix, :targets]in any permutation. Similarly for default values, when sources are loaded, and if a source is required to exist at: a certain time in: a certain env for: a certain target.
Thoughts? As far as I can tell, this DSL pretty much allows all compile-time config that actually impacts how a project’s dependencies generate source code to be provided exclusively by traditional config/*.exs mechanisms, and have all other current runtime-config usecases to be described by two single entries for config :runtime. This lets all actual runtime values live outside the config/*.exs world entirely, removing the need for any config/runtime.exs stuff, while preserving compile-time goodies furnished by the library itself.
Maybe it would be smarter to define a separate file like config_spec.exs and define all constraints there and just import it in your config.exs, in addition to this I’m not sure if you can, but I would definitely avoid using the config macro, since it is unclear whether you are defining a config value or a specification.
This is a bad thing in my opinion. I like elixir config because it is crystal clear how it works, all the base configs are evaluated at compile-time (this includes reading from local resources at compile-time) and runtime always loads at boot, with the system you are introducing, people should be a aware of possible limitations/quirks of your configuration lifecycle, witch introduces a lot of complexity into the configuration, witch is something I personally hate having to deal in other languages.
I do like @D4no0 's suggestion here of keeping the config’s semantically separate, with naming along the lines of config_spec.exs which I think is quite an aptly named file.
I’ll agree this is pain point; but mostly one of nomenclature.
Internally all the items in this list get built into something I’m calling a Runtime.Configuration.Specification.
Initially I tinkered with something like config :runtime_config, specifications: [...], sources: [...] as well, but that felt even more distanced from the fact that at some point we are producing a strongly-typed runtime :value. I’m wondering if you feel like that reads better, though.
I do feel like introducing new files like config_spec.exs makes things even more confusing in the config/config.exs file-naming story. Naming the :otp_app specifically :runtime and sourcing specifications inline via config, :runtime, values: [...] felt like a nice approach that reads better if you imagine a world where all runtime configuration is specified thusly, but I see where you’re coming from.
I do have to disagree here, I think it’s really unclear. We see posts to this effect all the time, even with experienced devs, in these forums (random recent example from a search) and especially from newcomers on stack overflow. That’s been a major motivation to me to ideate an alternate approach, whether or not it catches on.
I do feel like a LOT of pain could have been saved early on by simple renaming:
config/config.exs → config/compile_time.exsconfig/runtime.exs → config/boot_time.exsmaking it properly clear when these things get loaded, and that none of these configuration sources contain true, runtime-loadable, runtime-modifiable solutions (without System.get_env-deferring hacks and ultimately storing them in distributed-erlang-friendly datastores to support runtime modifications).
If I’m working within these constraints to build such a runtime-friendly system, I have to work within these historical decisions today, since Elixir 2.0 may never be published to support this kind of change. Having this be calls to config :runtime in a single config/config.exs file and a robust context DSL feels like the best way to do this to me.
Indeed, in my applications today, I tend to:
config_path: "config/compile_time.exs" in my mix.exsruntime_config_path: "config/boot_time.exs" in my mix.exsimport_config "#{config_env()}.exs" in my config/compile_time.exs, in favor of:
case config_env() do... in my config/compile_time.exscase config_target() do... in my config/boot_time.exs, ex for nerves projectsApplication callbacks with distributed Phoenix.PubSub and clever supervision treesThis is fine for myself, but spreads things even further around my codebase against convention, with less ability to reason about a project’s configuration in a single place.
Empowering devs to reason about this with a single library and a single declarative config/config.exs with the occasional case config_env() do... thrown in there for actual compile-time-required changes to how dependencies generate code in different envs is the major motivation of this project.
I also personally hate dealing with this complexity in other languages, which is why I’m trying to design something that can manage that complexity even at runtime-modification levels, but with sane defaults for the standard project requirements to support lightweight specification in common usage, that can all fit within a single config/config.exs file.
I know I’m refuting a lot of your feedback here, but I do want to make it clear that I really appreciate it, and it is indeed informing me about how this might be received even if I double down on my current approach. I think I may take a walk around the block for a bit, implement the distributed-system support and install it into my existing projects, and open-source things before I try to rethink if I want to handle the declarative configuration specification approach differently with your feedback in mind, armed with more context and real-world perspectives.
To dive into this deeper, I could avoid it, but that would prohibit me from doing my clever little compile-time guarantees as mentioned in my last post:
Pretty much the only window of time I have to modify how Runtime generates functions in different contexts pre-compilation based on the user’s input is by reading from Application.get_all_env(:runtime), so I’d have to say goodbye to some features without it.
Doing this via config has introduced some major pains in supporting user-defined custom Runtime.Configuration.Type and Runtime.Configuration.Source structs, since they aren’t yet available at Runtime’s compile-time, so abandoning it is not off the table. But I waited until I had a work-around for this and proven the concept before re-soliciting your opinions on the config soultion.
The actual complexity is the config impact with the metaprogramming system, it is true that it is rather peculiar that elixir was supporting only compile-time config back in the day, however your approach does not abstract it away, rather it tries to blend it in, adding another layer of possible complexity. It is also imperative that you should take into consideration why elixir design team decided to go with the runtime.exs approach, even though they had the possibility to make an implementation like yours. We all have our vision of how a system should work, based on what we worked/saw previously, however the philosophy of the language should be taken into consideration, otherwise we might end up with an abomination of language and ecosystem like javascript.
It is true that the default compile-time config could be somehow marked better, however the runtime.exs is named perfect, the config is loaded at runtime, the fact that it is loaded when the server boots is just a implementation detail. Back in the day, credo helped me a lot to understand clearly how configs should be used correctly, because it would throw an error when using Application.get_env on compile-time configs and force you to use Application.compile_env.
(I am not willing to describe to you how atrocious the Runtime.CompileTime module looks yet. 
)
I agree with this. I followed the discussion on the core mailing list and other forums while it was being developed, and I do think I have an intuition — they were trying to improve the compile-time-only issues of the time, to support boot-time configuration from ENV vars. Supporting runtime-modifiable configuration was way out of scope, and does belong in its own library. What upsets me is that trying develop such a library is immediately confusing by the name of runtime.exs instead of boot-time.exs, I wish I’d agitated more about the decision back then…