Specify: Comfortable, Explicit, Multi-Layered Configuration Specifications - v0.7

[Qqwy]

After two long topics (1 , 2) dedicated to how to best configure your elixir application,
talking with a couple of people at ElixirConf.eu about the current state of configuring applications,
and recognizing some patterns (both good and bad) in the Elixir libraries I and others have written in the last couple of years,

here is my take on the challenge of configuring applications and libraries in Elixir:

---

Comfortable, Explicit, multi-Layered configuration specifications.

hex - docs - github

Specify is an Elixir library to standardize the method of doing configuration for your applications and libraries. Rather than trying to force configuration to happen in one way, it enables both library writers and library consumers to specify on multiple different levels how they want to configure something.

The main goals that Specify has in mind are:

1. Configuration is based on looking at a stack of (also per-specification and globally configurable!) configuration sources, falling back to defaults written in the configuration specification.
2. Overriding the values in this stack of configuration using plain arguments passed to a function is also always possible, circumventing the ‘singleton problem’ that many configurable things suffer from.
3. It is very clear what values are supported for a given configuration field, since a validation/parsing function is named when specifying it.
4. Documentation is automatically generated from a configuration specification, listing names, descriptions, parser functions and default values.
5. Normalization: After a call tto YourConfig.load you end up with an Elixir struct containing the configuration, with all fields having passed validation and parsing, so you know exactly what fields are there to use and what kind of values to expect. This greatly reduces the number of weird ambiguous errors when something is misconfigured.
6. Fail-fast: The library raises errors on missing required configuration, things that cannot be parsed and attempting to override unexistent fields. The library raises compilation warnings on missing documentation. The library logs at ‘error’-log-level when a source cannot be reached during an attempted configuration load.

Example

defmodule Cosette.CastleOnACloud do
  require Specify
  Specify.defconfig sources: [Specify.Providers.MixEnv, Specify.Providers.Process] do
    @doc "there are no floors for me to sweep"
    field :floors_to_sweep, :integer, default: 0

    @doc "there are a hundred boys and girls"
    field :amount_boys_and_girls, :integer, default: 100

    @doc "The lady all in white holds me and sings a lullaby"
    field :lullaby, :string

    @doc "Crying is usually not allowed"
    field :crying_allowed, :boolean, default: false
  end
end

Trying to load this using Cosette.CatleOnACloud.load() will fail because a mandatory field is missing.

iex> Cosette.CastleOnACloud.load
** (Specify.MissingRequiredFieldsError) Missing required fields for `Elixir.Cosette.CastleOnACloud`: `:lullaby`.
    (specify) lib/specify.ex:179: Specify.prevent_missing_required_fields!/3
    (specify) lib/specify.ex:147: Specify.load/2

However, by configuring it using

Application.put_env(Cosette.CastleOnACloud, :lullaby, "I love you very much")
# or, in your config/conf.exs file:
config Cosette.CastleOnACloud, lullaby: "I love you very much"

or instead, using the Process Dictionary source:

Process.put(Cosette.CastleOnACloud, :lullaby, "I love you very much")

or specifying an inline lullaby value as part of the overrides: argument:

iex> Cosette.CastleOnACloud.load(overrides: [lullaby: "I love you very much", crying_allowed: true])

we end up with:

%Cosette.CastleOnACloud{
  crying_allowed: true,
  floors_to_sweep: 0,
  lullaby: "I love you very much",
  amount_boys_and_girls: 100
}

---

Roadmap

Specify is not yet done. Most importantly, it needs some tests and some more effort to make sure that its interface is understandable not only to me but also to the rest of the community .
Also, at the very least I still want to create an environment variables provider (both for .env-files and System.get_env).
I am fairly certain that its documentation and examples can be improved to be more understandable. Besides this, I would love to hear answers to the following practical questions:

Does Specify work with how you are used to doing configuration, or does it clash, and in what way?

I have followed along and read through both of afore-mentioned topics so I think/hope that Specify is able to be useful to most common use-cases. Of course, it is only a tool, and definitely is somewhat opinionated.
To make it as generally useful as possible, I very much would like to hear your suggestions/ideas/questions.

What would be the best way to read from environment variables?

Basically, how to deal with the mismatch of Elixir’s snake_case typing and environment variables that usually, but not necessarily, are in CONSTANT_CASE. I am currently thinking of adding additional options to the fields to define an environment variable name (which defaults to the uppercase field name, but is overridable), with an additional option to the defconfig call that will be prefixed to these names, but I am not yet entirely certain this is the best way.

All with all, it was a ton of fun to write Specify (it might have been my most involved macro- and metaprogramming in Elixir to date). Very eager for your feedback!

hex - docs - github

---

note: starting at v. 0.4.0, Confy has been renamed to Specify.

[OvermindDL1]

Took a bit to read it, looks like a fantastic start!

So a few starter questions:

1. You can make arbitrary configuration loaders and handlers?

2. What about one that pulls from a database on some kind of update message so it updates the configurations in real time, is there a way to tell something like a gen_server built on this to accept this new configuration (say like changing the pool size in a database pool)?

3. How do you differentiate between the different stages of configurations? I.E. Compile-time, Load-Time, and Run-Time (dynamic)?

[Qqwy]

@OvermindDL1

Took a bit to read it, looks like a fantastic start!

So a few starter questions:

1. You can make arbitrary configuration loaders and handlers?

Yes . The Confy.Provider protocol can be implemented by anything you want.

@OvermindDL1

2. What about one that pulls from a database on some kind of update message so it updates the configurations in real time, is there a way to tell something like a gen_server built on this to accept this new configuration (say like changing the pool size in a database pool)?

Currently, not. I am planning to, in a way similar to what Vapor does, offer the possibility to spin up a background ‘configuration change watcher’ process that will trigger a message when the configuration changes at runtime. (Unfortunately for most providers this will require polling).

The GenServer could then decide to switch over to the new configuration, stop and start a new version of itself with the new configuration, or potentially even recompile a module if, for optimization reasons, the compilation process itself is affected by the configuration.

@OvermindDL1

3. How do you differentiate between the different stages of configurations? I.E. Compile-time, Load-Time, and Run-Time (dynamic)?

Confy treats them in the same way. Configuration is only ever loaded when YourConfigModule.load(options \\ []) is called. At that time, Confy will look through the sources that you have configured, and for every field, decide which source has the highest priority.

So if you have the following configuration:

defmodule MyRepo do
  import Confy
  defconfig sources: [Confy.Providers.MixEnv, Confy.Providers.Process] do
    field :endpoint, :string, default: "localhost"
    field :port, :integer, default: 8086
  end
end

then when calling MyRepo.load(options) it will:

• Take the default values of :endpoint and :port
• Possibly override them by a value set in the Application configuration.
• Possibly override them by a value set in the current process’ process dictionary.
• Possibly override them by a value set in the overrides: key of the options passed to MyRepo.load.

---

The idea is that you call YourConfigModule.load once (or infrequently) and keep the struct it returns around in e.g. your GenServer state.This means that parsing of individual config fields has to be done only once, and that you end up with an explicit representation of the configuration. Also, because you end up with ‘just’ a struct, the logic that uses this configuration can be completely stateless and might be easier to test.

[OvermindDL1]

@Qqwy

(Unfortunately for most providers this will require polling).

I’d require the provider to handle that. For PostgreSQL for example you can LISTEN in it to get updates pushed for example.

The GenServer could then decide to switch over to the new configuration, stop and start a new version of itself with the new configuration, or potentially even recompile a module if, for optimization reasons, the compilation process itself is affected by the configuration.

The fact the OTP runtime doesn’t already have something that sends an update message to registered receivers when an application environment value changes is still a bit astounding to me… ^.^;

[Qqwy]

@OvermindDL1

I’d require the provider to handle that. For PostgreSQL for example you can LISTEN in it to get updates pushed for example.

That’s a good idea. The protocol could definitely be extended by such a function(which could be opt-in, falling back to longpolling) .

---

Today I managed to put some more work in. Important changes for v0.3.0 are:

• New load_explicit that takes first a list of explicit values you want to provide (and the second, optional, argument is the list of options).
• The :overrides key has been renamed to :explicit_values because it more clearly shows its purpose in my opinion.
• Lots of tests have been written, and a couple of bugs were ironed out.

I currently envision two use-cases for this library:

1. Parsing configuration that is to be used for a GenServer (or similar process), by calling MySettings.load() at process initialization time.
2. Parsing of the options passed to use (and similar macros or functions), by calling MySettings.load_explicit(options).

In both of these cases, the main advantage of using Confy is the explicitness:

• you (the code maintainer) know what fields exist
• you (the code maintainer) know what types these fields will be
• consumers (that call your code) have documentation to find out what options can be passed, and see readable compile-time (or start-up-time) errors when improper things are passed.

[Qqwy]

Starting today, Confy has been renamed to Specify to be more clear in the main focus of the library, which is to make fields and field-types explicit, regardless of whether we are talking about app-wide configuration, user-created settings or single-function-call options.

Specify v.0.4.0 has been released, which replaces Confy v.0.3.0.

Also, what is better than a beautiful library? A beautiful library that has its own logo!

[OvermindDL1]

@Qqwy

A beautiful library that has its own logo!

Ooo nice!

[Qqwy]

More tests and documentation has been written; version 0.4.2 has been released!

The library is inching towards a first stable release. There are still features that might be worthwhile to add, but the current core interface is starting to come together nicely.

[Qqwy]

Version 0.6 has been released!

Since the last update on the forum, the following improvements have been made:

• 0.5 - Adds the nonnegative_integer , positive_integer , nonnegative_float , positive_float and timeout builtin parsers.
• 0.6 - Adds the mfa and function builtin parsers.

---

I have one question to the community at large, which is about parsers like the function one. Currently there is no way to restrict a field expecting a function to a particular arity. What would be a good API for this?

I am thinking about two possibilities (but maybe there is a third as well):

• field(field_name, {:function, arity}, default: some_val)
• field(field_name, function(arity), default: some_val)

Currently, we already have {:list, :element_parser} (e.g. {:list, :integer}). It would be nice to keep a similar format, but I am not entirely convinced on the former format’s structure.

As for the roadmap: Currently the only thing that I’d consider blocking before v 1.0 would be the possibility to nest specifications. (And besides this resolving above issue, I guess).

[Qqwy]

Version 0.7 has been released!

This version adds:

• The possibility to make configuration source optional, which is rather nice because Specify warns when you do not fill in any fields in a particular configuration source. In certain cases it makes sense that a particular configuration source only exist some of the time (like setting optional system environment variables), which would make the warnings unneccesarily noisy.
• To make it easier to specify the default sources in e.g. the Mix.Config/Elixir.Config files where no dependencies (and thus not Specify either) are loaded yet, it now is possible to pass a source that you want to configure as {SourceModule, %{some: field, some_other: field}} which will be turned into a %SourceModule{some: field, some_other: field} struct on startup. A module/function/argument-syntax now is also provided, for even more flexibility.