Hm, I believe others are more helpful than me around here. I kinda stopped posting coding snippets because a lot of problems don’t interest me much (like all of Phoenix and LiveView).
Appreciate what you said. 
I suppose my ranting can be categorized as a commentary. 
It’s not something that’s purified, just some thoughts, plus I might start working sooner than I think and never get around to it, or I might do that plus improve my health and do it very soon… absolutely nothing is clear yet.
But yep, I’d like to think some more about it as well. It’s very important to clear up the value proposition before coding.
Having extracted out the initial library and having done some testing, persistent_term is clearly superior for this usecase, compared to my prior :ets solution; both in speed but also simplicity of implementation. I’m grateful to you reminding me about it early on in dev!
After some development, I’m interested in your thoughts with my prototype API. Today it looks akin to:
# config/config.exs
config :runtime, values: [
SIGNING_SALT: :string,
DATABASE_URL: :uri,
DATABASE_POOL: {:integer, min: 1, max: 100}
]
config :runtime, sources: [
json: [file: "priv/config.json"],
exs: [file: "priv/config.exs"],
dotenv: [file: "priv/config.env"],
env: []
]
While I’d love to distance myself even further from the config/*.exs world, having the library know these things in advance at compile-time lets me make very strong guarantees—for example:
Runtime.Config.get(:DATABAS_URL)
Runtime.Config.get(:DATABASE_URL) will always return a %URI{} struct and contrary usage should emit a warningThis API is Keyword based, so plays well with Config’s deep-merging strategy. This means theoretically you could still do stuff like:
# config/dev.exs
config :runtime, values: [
DATABASE_URL: [
type: :uri,
required: false,
default: {:value, "http://localhost:5432/my_app_dev"}
]
]
# config/prod.exs
config :runtime, values: [
DATABASE_URL: [type: :uri, required: true]
]
or:
# config/dev.exs
config :runtime, sources: [json: [file: "priv/config/dev.json"]]
# config/prod.exs
config :runtime, sources: [json: [file: "priv/config/prod.json"]]
However, the library would generally discourage such an approach, in favor of a more robust and consolidated “context” DSL. The above examples can be written as:
# config/config.exs
config :runtime, values: [
DATABASE_URL: [
type: :uri,
required: [in: :prod],
default: [in: [dev: {:value, "http://localhost:5432/my_app_dev"}]]
]
]
config :runtime, sources: [
json: [file: "priv/config/dev.json", in: :dev],
json: [file: "priv/config/prod.json", in: :prod]
]
The current context DSL further supports things like specifying that a value is required to have been provided in contexts:
at: :compile_time | :boot_time | :runtimein: [:list, :of, :mix, :envs],for: [:list, :of, :mix, :targets]in any permutation. Similarly for default values, when sources are loaded, and if a source is required to exist at: a certain time in: a certain env for: a certain target.
Thoughts? As far as I can tell, this DSL pretty much allows all compile-time config that actually impacts how a project’s dependencies generate source code to be provided exclusively by traditional config/*.exs mechanisms, and have all other current runtime-config usecases to be described by two single entries for config :runtime. This lets all actual runtime values live outside the config/*.exs world entirely, removing the need for any config/runtime.exs stuff, while preserving compile-time goodies furnished by the library itself.
Maybe it would be smarter to define a separate file like config_spec.exs and define all constraints there and just import it in your config.exs, in addition to this I’m not sure if you can, but I would definitely avoid using the config macro, since it is unclear whether you are defining a config value or a specification.
This is a bad thing in my opinion. I like elixir config because it is crystal clear how it works, all the base configs are evaluated at compile-time (this includes reading from local resources at compile-time) and runtime always loads at boot, with the system you are introducing, people should be a aware of possible limitations/quirks of your configuration lifecycle, witch introduces a lot of complexity into the configuration, witch is something I personally hate having to deal in other languages.
I do like @D4no0 's suggestion here of keeping the config’s semantically separate, with naming along the lines of config_spec.exs which I think is quite an aptly named file.
I’ll agree this is pain point; but mostly one of nomenclature.
Internally all the items in this list get built into something I’m calling a Runtime.Configuration.Specification.
Initially I tinkered with something like config :runtime_config, specifications: [...], sources: [...] as well, but that felt even more distanced from the fact that at some point we are producing a strongly-typed runtime :value. I’m wondering if you feel like that reads better, though.
I do feel like introducing new files like config_spec.exs makes things even more confusing in the config/config.exs file-naming story. Naming the :otp_app specifically :runtime and sourcing specifications inline via config, :runtime, values: [...] felt like a nice approach that reads better if you imagine a world where all runtime configuration is specified thusly, but I see where you’re coming from.
I do have to disagree here, I think it’s really unclear. We see posts to this effect all the time, even with experienced devs, in these forums (random recent example from a search) and especially from newcomers on stack overflow. That’s been a major motivation to me to ideate an alternate approach, whether or not it catches on.
I do feel like a LOT of pain could have been saved early on by simple renaming:
config/config.exs → config/compile_time.exs
config/runtime.exs → config/boot_time.exs
making it properly clear when these things get loaded, and that none of these configuration sources contain true, runtime-loadable, runtime-modifiable solutions (without System.get_env-deferring hacks and ultimately storing them in distributed-erlang-friendly datastores to support runtime modifications).
If I’m working within these constraints to build such a runtime-friendly system, I have to work within these historical decisions today, since Elixir 2.0 may never be published to support this kind of change. Having this be calls to config :runtime in a single config/config.exs file and a robust context DSL feels like the best way to do this to me.
Indeed, in my applications today, I tend to:
config_path: "config/compile_time.exs" in my mix.exs
runtime_config_path: "config/boot_time.exs" in my mix.exs
import_config "#{config_env()}.exs" in my config/compile_time.exs, in favor of:
case config_env() do... in my config/compile_time.exs
case config_target() do... in my config/boot_time.exs, ex for nerves projectsApplication callbacks with distributed Phoenix.PubSub and clever supervision treesThis is fine for myself, but spreads things even further around my codebase against convention, with less ability to reason about a project’s configuration in a single place.
Empowering devs to reason about this with a single library and a single declarative config/config.exs with the occasional case config_env() do... thrown in there for actual compile-time-required changes to how dependencies generate code in different envs is the major motivation of this project.
I also personally hate dealing with this complexity in other languages, which is why I’m trying to design something that can manage that complexity even at runtime-modification levels, but with sane defaults for the standard project requirements to support lightweight specification in common usage, that can all fit within a single config/config.exs file.
I know I’m refuting a lot of your feedback here, but I do want to make it clear that I really appreciate it, and it is indeed informing me about how this might be received even if I double down on my current approach. I think I may take a walk around the block for a bit, implement the distributed-system support and install it into my existing projects, and open-source things before I try to rethink if I want to handle the declarative configuration specification approach differently with your feedback in mind, armed with more context and real-world perspectives.
To dive into this deeper, I could avoid it, but that would prohibit me from doing my clever little compile-time guarantees as mentioned in my last post:
Pretty much the only window of time I have to modify how Runtime generates functions in different contexts pre-compilation based on the user’s input is by reading from Application.get_all_env(:runtime), so I’d have to say goodbye to some features without it.
Doing this via config has introduced some major pains in supporting user-defined custom Runtime.Configuration.Type and Runtime.Configuration.Source structs, since they aren’t yet available at Runtime’s compile-time, so abandoning it is not off the table. But I waited until I had a work-around for this and proven the concept before re-soliciting your opinions on the config soultion.
The actual complexity is the config impact with the metaprogramming system, it is true that it is rather peculiar that elixir was supporting only compile-time config back in the day, however your approach does not abstract it away, rather it tries to blend it in, adding another layer of possible complexity. It is also imperative that you should take into consideration why elixir design team decided to go with the runtime.exs approach, even though they had the possibility to make an implementation like yours. We all have our vision of how a system should work, based on what we worked/saw previously, however the philosophy of the language should be taken into consideration, otherwise we might end up with an abomination of language and ecosystem like javascript.
It is true that the default compile-time config could be somehow marked better, however the runtime.exs is named perfect, the config is loaded at runtime, the fact that it is loaded when the server boots is just a implementation detail. Back in the day, credo helped me a lot to understand clearly how configs should be used correctly, because it would throw an error when using Application.get_env on compile-time configs and force you to use Application.compile_env.
(I am not willing to describe to you how atrocious the Runtime.CompileTime module looks yet. 
)
I agree with this. I followed the discussion on the core mailing list and other forums while it was being developed, and I do think I have an intuition — they were trying to improve the compile-time-only issues of the time, to support boot-time configuration from ENV vars. Supporting runtime-modifiable configuration was way out of scope, and does belong in its own library. What upsets me is that trying develop such a library is immediately confusing by the name of runtime.exs instead of boot-time.exs, I wish I’d agitated more about the decision back then…
Trying to build something that supports validated modification of configuration values at runtime, well after boot-time, makes it clear how there are really 3 different contexts: configuration that is required at compile-time, configuration that is read at runtime, and the special case of runtime configuration that should be required for the application to boot successfully.
To support strongly-typed, validatable configuration values, that can be modified post-boot at runtime, I’ve been wanting to separate the boot-time special-case into its own discrete concept, so that I can perform validations of said configuration within the Runtime application’s boot cycle, to fail fast instead of waiting for said values to be read at a later point in runtime and only then be discovered to be invalid.
For example, I’ve had Elixir, Ruby, and Python applications where something like the equivalent of a ERROR_REPORTING_SYSTEM_CALLBACK_URL was mis-configured as an invalid url in certain environments. The applications booted happily and ran for days until trying to report an error, and only ate dirt when trying to use the mis-configured value at runtime (with the exception of the Elixir application, which recovered thanks to supervisors, but delayed discovery of the issue for weeks). Boot-time validation of that runtime configuration value would have saved me many times. When the error reporting url is mis-configured, you’re deeply screwed.
This is the sort of very specific issue I’m trying to provide tooling address, but I agree the juice is not always worth the squeeze for simpler applications. I do like the idea of providing a solution to the config/when-the-hell-am-i-loaded-or-modifiable.exs confusion along the way for even simple applications and newcomers by allowing consolidating into config/config.exs alone along the way, though.
This is true, what I’m trying to achieve is a third layer: true modifiable-at-runtime, distributed, post-compile post-boot configuration. That needs complicate things. I would like to subsume the complexity of the other two existing options along the way to offer a unified experience, though.
Hi @christhekeele! I wanted to drop by and say that I’ve been eagerly following your progress on this library. I do think that this is touching a need for some applications and I’m curious to see what you come up with.
Once you’ve released a semi-stable version I’ll look into adding it to some of my side projects.
My only feedback at this time is that naming the application/library :runtime might confuse some users into thinking that a line like:
config :runtime, values: [
SIGNING_SALT: :string,
DATABASE_URL: :uri,
DATABASE_POOL: {:integer, min: 1, max: 100}
]
Is something built directly into Elixir instead of a separate library. Whereas giving the application a more unique name wouldn’t cause that same sort of confusion.
I’ve been going back and forth between RuntimeConfig, and Plasma — in homage to Vapor, as another cool state of matter.
I’ll mention dotenvy here because it is relevant to the 12-factor app ideas and it attempts to address some of the pain points that can crop up with Elixir configuration. See also the related article.
I hadn’t found dotenvy before! I really like what you’re doing with it.
config/config.exs, to get a unified story for non-dotenv sources and metaprogram certain config key access guarantees, so I don’t think I can leverage its source-loading tooling.cool – hope it’s helpful! I haven’t tried, but I think you could use the env!/3 function when dealing with compile-time config in config/config.exs to read system ENVs and take advantage of the type-casting. There’s no need to read/source .env files if you’re dealing with system ENVs that are already present. And yeah, the parser is pretty self-contained, so feel free to copy it. I’d considered publishing it as a stand-alone package, but I opted to avoid the deps. I hadn’t thought about it, but I don’t think there’s anything that would prevent one from using dotenvy to deal with compile-time config… even though the inspiration for it came from dealing with runtime config. 
Right, I allow the same config/source specifications to be loaded at either compile time, boot time, or runtime as appropriate, so I am doing most of my stuff to support that at compile time.
© Copyright Elixir Forum | Terms | Privacy & Cookies
