TL;DR:
I’m planning on building a library inspired by Vapor
to make configuring straight-forward Elixir applications easy; in a way that scales such that, when configuring your application becomes much less straight-forward, the task is much easier. Interested?
Backstory
I know there are quite a few options out there, and a lot of recent developments in this space the last few years, but I’ve still found building a good runtime configuration story in Elixir for large 12-factor apps to be a bit of a pain.
The problem is mostly permutative: over time, an application can grow to want multiple sources of configuration; modified differently at compile-time, boot-time, or runtime; differently in different build environments and targets; with support for different configuration file types; and with different approaches for overriding values during development and testing. This gets harder and harder to reason about without good developer tooling. Sound painfully familiar? Read on!
I’ve been porting a personal Elixir solution from project to project over the last 5 years, starting from the excellent Vapor
library, pretty much since the day it was released. As my pet approach has evolved, I’m pretty happy with it, but would love to polish it—and I’m tired of copy-pasting my own code again and again.
Some of this approach was stolen from my Ruby on Rails days, where I was equally dissatisfied with the situation in that ecosystem, and developed a similar personal non-open-sourced solution that worked well with Rails’ boot system and Ruby’s dynamacism, around which I built the Inquisitive gem for even more Ruby syntax-sugary ways of interacting with runtime configuration, first deployed in production applications around a decade ago.
Elixir, as a less-dynamic-than-Ruby, compiled language with a (historically driven by erlang release mechanisms) mostly build-time configuration story, has a harder-to-engineer story around runtime configuration. There’s been a lot of improvements to this in the first decade of Elixir, but there are still pain points.
I’m planning on codifying my Elixir approach to this problem in a package anyways for personal convenience, but I’m curious if there’s wider interest in the community, and would like to solicit ideas for a feature roadmap that might gel with what I’m building!
Synopsis
What I’m developing is essentially a declarative way to define your runtime configuration, and integrate it into your project at any point in your application’s development lifecycle.
Vapor’s example shows you how to throw it in to your Application.start/2
; but I unerringly find myself wrapping that in complex conditionals and OTP conveniences as the development, deployment, and override configuration-sophistication needs of my projects increases; always re-evolving the implementation towards the same result.
I figured it might be beneficial to encode my approach as a library, and make it easy for folks other than myself to re-use. Here’s what I have in mind:
Core Features
These are aspects of this system I’ve actually built before, and would love to stop re-inventing:
-
Support multiple approaches to defining configuration and sources:
- A straightforward
config.exs
-driven approach. - An inline-
Supervisor
-tree-driven approach (including, your main OTPApplication
supervisor, as the Vapor docs guide you towards). - Perhaps a module-driven approach DSL approach
- A straightforward
-
Config
env and target aware filters in configuration plans:To make it easier to describe a complicated permutation of sources for configuration values in different build environments and targets.
For example: loading from
.env.#{Config.config_env()}
-type files, but never when deployed to:prod
, where the app should rely exclusively on environment variables. Or, looking into a.gitignore
d.env.local
configuration file for ultimate overrides, but never doing so outsideMIX_ENV=dev
orMIX_TARGET=local
situation.Specifically, instead of repeating configuration in different
config/#{Config.config_env()}.exs
files, allowing a single source of truth entry in your mainconfig.exs
file, with filters attached (similar to yourmix.exs
deps()
:env
and:targets
filters). This makes it much easier to reason about where your runtime configuration comes from in your build-time configuration.Or, describing a single list of configuration providers in your
Application.start/2
callback, instead of incrementally building a list with manyproviders = if Config.config_env() == desired_env, do: modify_providers_for_this_permutation(providers)
calls -
A handful of trivial out-of-the-box mappers for common config coercions:
Ex: modeling string-only env vars as booleans, ints, or floats at runtime.
-
A validation system for ensuring values are within required parameters:
Ex: ensuring that your database pool size is always greater than 1 in production.
-
Very specific error messages when required configuration values are missing or cannot parse:
Including a lineage of all config sources that attempted to provide a value.
-
A Mix task for ensuring configuration is loaded appropriately for other mix tasks:
Necessary when your config loading is done externally to your
Application.start
call.For example, if your Ecto Repo uses the
init/2
callback to configure itself dynamically at runtime,mix ecto...
will not work without a little help in accessing config not loaded in your main application callback, if it is provided by libraries such as these. -
A Mix task for easy introspection of the current configuration given the current env/target, including lineage of overrides from different configuration sources.
-
Logger
output atApplication
startup about configuration values:To make it trivial to understand in your logs the way in which your application was configured at launch, including override lineage.
-
Secret-awareness to prevent sensitive things from being logged or displayed in Mix tasks.
Aspirational Features
Features for this system I’ve never implemented before, but believe I can build it to support, with enough motivation:
-
An extensible system of declaring configuration value parser Vapor “mapping” functions:
-
Working around restrictions in referencing anonymous functions in a
config.exs
, to support all usage modes.Generally by the time I find this need, I’ve moved configuration over into my
Application.start/2
callback where they are already available, but an out-of-the-box solution that supportsconfig.exs
configuration as a first-class citizen must accommodate this.
-
-
Test helpers, to make overwriting runtime config during a test (and other mocking of configuration values) trivial without fully losing parallelization of said tests:
Regardless of configuration value providence, like an environment variable that is hard to modify mid-test-suite, this would let you play nicely with the virtuous properties of
ExUnit
. -
Per-process caching of commonly fetched config values in the process dictionary for hot paths and tight loops (since the initial library plan currently throws everything into
:ets
for retrieval at runtime each time it is referenced, and would only grow less performant with distributed-friendly alternative implementations of the configuration backend). -
Swappable backends over
:ets
to extend the configuration value storage mechanism to more distributed-friendly environments, once I am convinced we can optimize this scenario for hot paths and updates. For example, anyEcto
-supported adapter, orpersistent_term
for scenarios where configuration is rarely intended to be changed. -
Sane support for changing runtime configuration values at runtime, even with distributed backends, so that it is viable to do so via a remote connection to a production system:
-
With a Pub-Sub system for configuration value consumers to be notified when this occurs.
-
And supervision tree helpers subscribed to that to make restarting when certain values change trivial, ex:
[ {Library.Configuration.Dependency, values: [:SECRET_KEY_BASE, :SECRET_SALT]}, MyApp.Endpoint ] |> Supervisor.start_link(strategy: :rest_for_one, name: MyApp.WebSupervisor)
or even
[ { Library.Configuration.Watcher, values: [:SECRET_KEY_BASE, :SECRET_SALT]}, children: [MyApp.Endpoint] , name: MyApp.WebSupervisor }, Other.Things ] |> MyApp.Supervisor.start_link(strategy: :one_for_one)
letting you literally connect to a running production application and rotate your secret keys, at runtime, with zero downtime outside of your
Supervisor
s restarting things.Or more generally, modifying any configuration in a production system at runtime that you’ve decided to make runtime configuration, with OTP supervision tree resiliency guarantees about the consequences.
-
Call for feedback, criticism, and ideas
Does any of this excite you, or feel like it might solve a pain point in the projects you work on? Let me know!
Or, do you maintain a complicated and large 12-factor app, and this still seems over-engineered and unrealistically overblown—would you loathe working in a system configured this way?
Finally, this is all conceived from my own personal experience, needs, and observing those of others here on this forum. Do you have any other insights from your experience you think would be instructive during the initial development of such a library?
Thanks for reading! Let me know your thoughts!