TL;DR:

I’m planning on building a library, inspired by Vapor, to make configuring Elixir applications more straight-forward; in an approachable but highly flexible way that scales to non-trivial usecases, even supporting no-redeploy modification of runtime configuration values in distributed, hot-code-reloaded systems. Interested?

Summary

I know there are quite a few options out there, and a lot of recent developments in this space the last few years, but I’ve still found building a good runtime configuration story in Elixir for large 12-factor apps to be a bit of a pain.

The problem is partially permutative: over time, an application can grow to want multiple sources of runtime configuration; loaded differently at compile-time, boot-time, or runtime; differently in different build environments and targets; with support for different configuration file types; and with different approaches for overriding values during development and testing. This gets harder and harder to reason about without good developer tooling.

Add in the ability to modify these values at runtime, in distributed systems, that supports hot-code reloading, and the problem becomes nearly intractable. I’d like to tract it.

Backstory

I’ve been porting a personal Elixir solution from project to project over the last 5 years, starting from the excellent Vapor library, pretty much since the day it was released. As my pet approach has evolved, I’m pretty happy with it, but would love to polish it—and I’m tired of copy-pasting my own code again and again.

Some of this approach was stolen from my Ruby on Rails days, where I was equally dissatisfied with the situation in that ecosystem, and developed a similar personal non-open-sourced solution that worked well with Rails’ boot system and Ruby’s dynamacism, around which I built the Inquisitive gem for even more Ruby syntax-sugary ways of interacting with runtime configuration, first deployed in production applications around a decade ago.

Elixir, as a less-dynamic-than-Ruby, compiled language with a (historically driven by erlang release mechanisms) mostly build-time configuration story, has a harder-to-engineer story around runtime configuration. There’s been a lot of improvements to this in the first decade of Elixir, but there are still pain points.

I’m planning on codifying my Elixir approach to this problem in a package anyways for personal convenience, but I’m curious if there’s wider interest in the community, and would like to solicit ideas for a feature roadmap that might gel with what I’m building!

Synopsis

What I’m developing is essentially a declarative way to define your runtime configuration, and integrate it into your project at any point in your application’s development lifecycle.

Vapor’s example shows you how to throw it in to your Application.start/2; but I unerringly find myself wrapping that in complex conditionals and OTP conveniences as the development, deployment, and override configuration-sophistication needs of my projects increases; always re-evolving the implementation towards the same result.

I figured it might be beneficial to encode my approach as a library, and make it easy for folks other than myself to re-use. Here’s what I have in mind:

Core Features

These are aspects of this system I’ve actually built before, and would love to stop re-inventing:

• Support multiple approaches to defining configuration and sources:

  • A straightforward config.exs-driven approach.
  • An inline-Supervisor-tree-driven approach (including, your main OTP Application supervisor, as the Vapor docs guide you towards).
  • Perhaps a module-driven approach DSL approach

• Config env and target aware filters in configuration plans:

  To make it easier to describe a complicated permutation of sources for configuration values in different build environments and targets.

  For example: loading from .env.#{Config.config_env()}-type files, but never when deployed to :prod, where the app should rely exclusively on environment variables. Or, looking into a .gitignored .env.local configuration file for ultimate overrides, but never doing so outside MIX_ENV=dev or MIX_TARGET=local situation.

  Specifically, instead of repeating configuration in different config/#{Config.config_env()}.exs files, allowing a single source of truth entry in your main config.exs file, with filters attached (similar to your mix.exs deps() :env and :targets filters). This makes it much easier to reason about where your runtime configuration comes from in your build-time configuration.

  Or, describing a single list of configuration providers in your Application.start/2 callback, instead of incrementally building a list with many providers = if Config.config_env() == desired_env, do: modify_providers_for_this_permutation(providers) calls

• A handful of trivial out-of-the-box mappers for common config coercions:

  Ex: modeling string-only env vars as booleans, ints, or floats at runtime.

• A validation system for ensuring values are within required parameters:

  Ex: ensuring that your database pool size is always greater than 1 in production.

• Very specific error messages when required configuration values are missing or cannot parse:

  Including a lineage of all config sources that attempted to provide a value.

• A Mix task for ensuring configuration is loaded appropriately for other mix tasks:

  Necessary when your config loading is done externally to your Application.start call.

  For example, if your Ecto Repo uses the init/2 callback to configure itself dynamically at runtime, mix ecto... will not work without a little help in accessing config not loaded in your main application callback, if it is provided by libraries such as these.

• A Mix task for easy introspection of the current configuration given the current env/target, including lineage of overrides from different configuration sources.

• Logger output at Application startup about configuration values:

  To make it trivial to understand in your logs the way in which your application was configured at launch, including override lineage.

• Secret-awareness to prevent sensitive things from being logged or displayed in Mix tasks.

Aspirational Features

Features for this system I’ve never implemented before, but believe I can build it to support, with enough motivation:

• An extensible system of declaring configuration value parser Vapor “mapping” functions:

  • Working around restrictions in referencing anonymous functions in a config.exs, to support all usage modes.

    Generally by the time I find this need, I’ve moved configuration over into my Application.start/2 callback where they are already available, but an out-of-the-box solution that supports config.exs configuration as a first-class citizen must accommodate this.

• Test helpers, to make overwriting runtime config during a test (and other mocking of configuration values) trivial without fully losing parallelization of said tests:

  Regardless of configuration value providence, like an environment variable that is hard to modify mid-test-suite, this would let you play nicely with the virtuous properties of ExUnit.

• Per-process caching of commonly fetched config values in the process dictionary for hot paths and tight loops (since the initial library plan currently throws everything into :ets for retrieval at runtime each time it is referenced, and would only grow less performant with distributed-friendly alternative implementations of the configuration backend).

• Swappable backends over :ets to extend the configuration value storage mechanism to more distributed-friendly environments, once I am convinced we can optimize this scenario for hot paths and updates. For example, any Ecto-supported adapter, or persistent_term for scenarios where configuration is rarely intended to be changed.

• Sane support for changing runtime configuration values at runtime, even with distributed backends, so that it is viable to do so via a remote connection to a production system:

  • With a Pub-Sub system for configuration value consumers to be notified when this occurs.

  • And supervision tree helpers subscribed to that to make restarting when certain values change trivial, ex:

     [
       {Library.Configuration.Dependency, values: [:SECRET_KEY_BASE, :SECRET_SALT]},
       MyApp.Endpoint
     ] |> Supervisor.start_link(strategy: :rest_for_one, name: MyApp.WebSupervisor)
    

    or even

    [
      {
        Library.Configuration.Watcher,
        values: [:SECRET_KEY_BASE, :SECRET_SALT]},
        children: [MyApp.Endpoint] ,
        name: MyApp.WebSupervisor
      },
      Other.Things
    ] |> MyApp.Supervisor.start_link(strategy: :one_for_one)
    

    letting you literally connect to a running production application and rotate your secret keys, at runtime, with zero downtime outside of your Supervisors restarting things.

    Or more generally, modifying any configuration in a production system at runtime that you’ve decided to make runtime configuration, with OTP supervision tree resiliency guarantees about the consequences.

Call for feedback, criticism, and ideas

Does any of this excite you, or feel like it might solve a pain point in the projects you work on? Let me know!

Or, do you maintain a complicated and large 12-factor app, and this still seems over-engineered and unrealistically overblown—would you loathe working in a system configured this way?

Finally, this is all conceived from my own personal experience, needs, and observing those of others here on this forum. Do you have any other insights from your experience you think would be instructive during the initial development of such a library?

Thanks for reading! Let me know your thoughts!