Sounds very promising, my only 5 cents is that I think it would be wise to make this configuration type as separate as possible from the classic elixir configuration, as to not introduce more complexity into the runtime and compile-time configuration bucket.
1 Like
Agreed. My preference over time has been to relegate config.exs files exclusively to compile-time config, and placing all runtime-config closer to the Application.start/2 callback boot constructs, dodging the nuances between config/runtime.exs and others completely. It’s nice not having to have a runtime.exs file outside of Nerves projects, makes it much more intuitive to reason about!
But, I would like a full-solution library to support an ease-of-installation-accommodating the config.exs approach for convenience in new projects, which requires some work-arounds with what’s possible with Vapor, making it even more useful to abstract behind a library!
2 Likes
A solution for a distributed database for the configuration may be LiteFS - Distributed SQLite:
LiteFS is a distributed file system that transparently replicates SQLite databases. This lets you run your application like it’s running against a local on-disk SQLite database but behind the scenes the database is replicated to all the nodes in your cluster. This lets you run your database right next to your application on the edge.
1 Like
I’ve been really excited for the renaissance of SQLite in non-mobile, distributed production deployments, for exactly this sort of use-case, and fly.io is really supportive for this type of tech right now!
I’ll admit, I am leery of building this sort of library (initially) around a backend-storage swappable-adapter model (partially because of my experience trying to do so non-trivially with Mnemonix); just because of hot-path performance implications in the domain of configuration value reading. I stopped developing Mnemonix when I drew some flamegraphs around my first real-world applications using it, and read the writing on the wall about how my OTP-driven adapter architecture would throttle meaningful performance within the correct level of abstraction.
However, I agree that such an architecture would open up the doors to many distributed system setups! One more reason why I want to tackle this as a library instead of a repeated copy-paste hack: so I can properly encode the correct level of abstraction for this domain. In my analysis, the requirements of a ready-heavy runtime-configuration-reader library with good event-driven cache-busting is far more amenable to optimization than Mnemonix ever could have been.
My long-form aspirations here are to:
- Get things working first (via
:ets). - Provide a solution for hot paths next (via process-level caching and event-driven cache-busting).
- Not mentioned in my initial roadmap, finally return to the codebase with my experience from
Mnemonixand Elixir library development since then:- Specifically to support this anticipated abstraction of the storage level, when I’m confident that hot paths have a way to keep up.
I’m pretty delighted that LiteFS, Ecto.Adapters.SQLite3, and Etso have converged to a point of maturity around the same time, honestly. Between that, and some of @lawik’s recent analysis of distributed PG ↔ SQLite synchronization tools that would enable this library to work in a distributed fashion for feature flags—well, it’s just an exciting time to be an Elixir developer, and that’s a large part of what’s been making me itch to encode this as a robust library!
4 Likes
@Exadra37 I’ve added support for an alternative configuration backend over :ets, explicitly mentioning support for distributed environments, to the post’s Aspirational goals, because you’ve convinced me that this could be a major addition to the project, outside of its Core MVP goals!
1 Like
Coming from a dynamic language background the Elixir configuration was a pain to grasp and remember each time I came back to Elixir, worst when I started to deploy my pet apps, thus I really welcome a library that can make it easy to work with configuration and not hard to remember when returning back to the project after a while away.
As a developer advocate for security I don’t recommend at all that releases are built with any type of secrets on them, has we usually do now, with the session salt and session encryption key being a good example of some not being easy/possible to retrieve only at boot-time. It would be nice you could solve this problem with you configuration library.`
Maybe you want to keep an eye on Castle and/or work with them to be compatible with how configuration works with Hot Code Upgrades:
For example, to be compatible with sys.config :
runtime support for sys.config generation (incl. support for runtime.exs)
4 Likes
Same btw, and every time I had to configure :cowboy SSL I made a mistake. Configurations are not strongly typed nor enforced so any small mistake you only find out in runtime. Really started being a thorn in my butt for some time now.
I am pondering a different (smaller) library that wraps various common configurations in strongly-typed structs with clear rules which key must exist and when (f.ex. if you have one key present then two others are unnecessary, or if you put one optional key in then 3 others become mandatory because all 4 together must configure a certain aspect etc.) – and then they’ll translate these structs to the underlying mish-mash of [keyword] lists and tuples.
Would you have interest in that?
I am not even sure I’ll come back to work for an Elixir company, though I have started getting offers lately.
But if I don’t go all-in with Rust and do remain with Elixir on a part- or full-time job capacity then I very likely might end up writing such a library, just out of frustration.
4 Likes
If you’re concerned about ETS being slow, have you considered using persistent_term?
1 Like
Maybe you want to keep an eye on Castle and/or work with them to be compatible with how configuration works with Hot Code Upgrades
sys.config generation is really the main thing Castle does, both at boot time and (just prior to) hot-upgrade time. This satisfies the erlang release handler. Today, that generation only supports runtime.exs but more general support for Config Providers will be added shortly. As long as other providers implement the Config.Provider behaviour, Castle will be able to call them.
That isn’t the tricky bit tho - the tricky bit is correctly implementing the config_change callback in your application.
1 Like
I’m not too concerned about ETS being slow! And persistent_term would be a better fit for consumers who intend to not update runtime configuration that much, so it makes to support as an alternative backend. Will add to the candidate list for later feature development.
Exposing per-process caching in the process dictionary is more something I think I’ll be implementing anyways for library internals, so might choose to expose as a feature. In order to develop test helpers that let you run tests concurrently, but override some values for specific tests, I need a mechanism to let individual test processes access the override safely without modifying global configuration, and was thinking about using a read-through cache from process dict to configuration singleton.
If I don’t make that read-through cache only happen in test environments, but make that how the library does all lookups, then it’s trivial to expose what I suspect is the fastest possible way to optimize access to config values —so might as well expose if the implementation seems sound and compatible with other internals.
2 Likes
While I used Ecto database configuration as my initial example, I have used this tech specifically to have my session salts and encryption keys be runtime-only! They’re why I began looking into sane mechanisms for making updating such runtime configuration easy, as a zero-deployment solution to rotate the session encryption key in the event of a breach, and as a way to modify the session salt to force site-wide logout for everyone.
In fact this is a much nicer example, so I’ve updated to use it accordingly.
So yes, this library should solve this problem, as it already has before! IIRC it took a little more finagling then one’d like, but as this blog post about using vapor describes, both Ecto.Repo and Phoenix.Endpoint support init/1 callbacks now so either can be used with runtime config!
1 Like
That’s a great description, but in my opinion you’re trying to solve problem which is not actually that important for the ecosystem. During my engineering experience, I’ve encountered completely different problems with runtime configuration, like
- Distributed application configuration
- Atomic reconfiguration (the configuration is changed in multiple places “at once”)
- Configuration which has to perform some action to configure the different states
Generally speaking, I find every runtime configuration-as-a-key-value-store approach really hard to maintain, because configuring application in runtime is not a problem of changing a value, but really is a problem of propagating the changed value, and at most of the times this propagation must be as much as consistent and as atomic as possible.
OTP’s Application env is only applicable during initialization of the program, while it provides no meaningful answers or tools for runtime reconfiguration. Vapor actually solves a problem with different configuration stores and provides a DSL’s where plain Elixir could’ve been used without any implications or drawbacks.
So, if I was up to writing a runtime configuration library, I would have started with thinking about approaches to these problems.
To get an idea of what I am talking about, here’s an example:
Short example: we read the value from somewhere during the initialization and store it in the persistent_term, for faster read access in runtime. How do we change this value in runtime? Application.put_env is not working. We need hooks for reconfiguration or something like this.
Long example: we have a pool of workers around of TCP connections which are always connected to the server. They reconnect every time server drop’s the connection, send empty ACK’s, etc.
The server’s address is read from configuration (via application env, system env or vapor, it doesn’t matter) and then this address is stored in the state of each of these worker processes.
Problem: I want to reconnect to another address with the same state workers have right now. This can be done due to security reasons, or as a connection to fallback server in case of main server failing or whatever.
Existing configuration solutions provide to easy way to do this. What I actually need to do, is to perform a transaction which would suspend the workers, stop their connections, swap addresses in their states, initiate new connection and resume the workers. If something fails, I would have to rollback the actions and return to the user that the reconfiguration has failed.
Hard, right? Now imagine this pool is distributed
So, if I was up to writing a configuration library, I would start with adopting existing places where developers usually put the values they’ve read from configuration like persistent_term, ets, GenServer’s state. Next thing I would do, I would think about some configuration hook system, or something like this to have actions running when the values in the store are changed. And the last thing, I would provide some interfaces for tracking the transactional reconfigurations
1 Like
That’s the problem I am talking about. Configuration values stored in pdict make them almost impossible to reconfigure in runtime with existing tools
1 Like
It seems our heads are in the same place:
I think you’re gonna love where I want to take this project, then! Admittedly, I’m targeting something smaller initially, but if you read through the aspirational goals, you’ll notice these are the problems I’m reaching towards. I think I’ve found the right level of core abstraction that is extensible to solve these problems, and want to ship that first—we’ll see how it goes from there!
1 Like
Agreed. This technique is required for how I think I’ll be implementing test helpers, in a “block only” format that always tears down its own pdict overrides afterwards, but if I expose it as a library feature for production usage, it’d probably be in something like an Advance Usaged section, with ample caveats, such as you REALLY should ensure the process in question lives in a receive loop and checks its mailbox for cache-busting events from the library.
I think that’d pair well with what I’m working on, but uncertain if I’d adopt it within the library, though.
Vapor itself has a notion of individually required values, and a value-mapping feature that, if you use it assertively with functions that raise, lets you get pretty far at weeding out bad input from the system.
The other extreme would be supporting coercion into Ecto-esque schemas for configuration values (embedded schemas too! remember, config values are not necessarily single terms, they can be data structures if loading from ex. json or yaml files), or even something constraint-solver-esque like what you are describing, that models requirements between values.
The sweet spot for me, personally, would be a thin layer on top of what Vapor is doing to let typechecking warn for invalid config-value-lookup usage. Ex, static analysis tools should know if Configuration.lookup(:DATABASE_POOL_SIZE) returns a string or integer, and warn if used in a function known to require something different, since this is one of the large pains of env var/dotenv configuration loaders in collaborative projects: they are “stringly-typed”, and a developer looking up a value may not be certain what coercions have been applied, especially if the value could also come from other filetypes with stronger typing notions.
This is definitely something I want to think about a little later, but definitely something I want to think about more!
1 Like
Incidentally, while I understand and respect this sentiment, I would miss your presence and commentary on these forums!
1 Like
Hm, I believe others are more helpful than me around here. I kinda stopped posting coding snippets because a lot of problems don’t interest me much (like all of Phoenix and LiveView).
Appreciate what you said. 
I suppose my ranting can be categorized as a commentary. 
1 Like
It’s not something that’s purified, just some thoughts, plus I might start working sooner than I think and never get around to it, or I might do that plus improve my health and do it very soon… absolutely nothing is clear yet.
But yep, I’d like to think some more about it as well. It’s very important to clear up the value proposition before coding.
2 Likes
Having extracted out the initial library and having done some testing, persistent_term is clearly superior for this usecase, compared to my prior :ets solution; both in speed but also simplicity of implementation. I’m grateful to you reminding me about it early on in dev!
