Runtime configuration library (with casting, validation etc...) for native releases

We’re using Vapor extensively at my work: GitHub - elixir-toniq/vapor: Runtime configuration system for Elixir · GitHub.

The README probably needs to be touched up a bit. We’ve added a lot of ergonomics (specifying translations with a config value, grouping providers, etc.) that aren’t well documented atm.

Yeah, we’ll opensource that lib, though I’m not exactly sure when. I’ll try to make it happen soon-ish

As a bit of a background, when I wrote that lib, I took a glance at vapor (I wasn’t aware of skogsrå at the time, and I only learned of specify in your post ). Perhaps it was just the lack of docs, but my impression was that for the stuff we wanted, we’d have to implement most of functionality ourselves. Since my clients currently only use OS env, the conclusion was that vapor would basically be used as a glorified System.env wrapper, so we didn’t end up using it.

As demoed in those tweets, here are some of the things we wanted:

1. The ability to automatically create operator templates (.env files) automatically. This is an absolute must for us, because without it producing the list of required & optional vars is extremely error prone.
2. Providing sane defaults in dev/test allows us to make the projects runnable out of the box. As long as you’re e.g. running postgresql on the default port, the code will work without any special configuration. In particular, we don’t need to maintain a set of separate .env files or use envrc (which was also error prone, leading to frequent situations where project works for one dev, but doesn’t for others).
3. Compile-time generation of access functions makes it possible to detect spelling errors (e.g. db_poll_size) during compilation.
4. Included typespecs improve reasoning about the code.

All this being said, I feel that we don’t need such fragmentation, and that it would be better if we somehow rallied behind a single implementation. If other authors are open to this idea, I’m up for discussing it further.

But in any case, I feel we definitely need some solutions which make operator configuration simple, so in that sense, I guess it’s better to have four libs instead of zero

More likely these features just didn’t exist yet. Vapor was pretty bare bones. We’ve added more ergonomics based on feedback from the other engineers at B/R. But a lot of those haven’t been documented in the README yet.

Vapor allows you to provide defaults for a binding. But we typically recommend people don’t do that unless they have to. Defaults are only for convenience and will eventually lead to bugs in production. I was initially of the opinion that Vapor shouldn’t support defaults at all but was eventually overruled. In order to support a nice dev experience, we check in the .env files to our repos. Vapor’s dotenv provider supports a hierarchy of files and one of those is .env.local. This is where developers will overwrite values or add tokens, etc. We ensure *.local files are gitignored so they don’t get checked in.

I meant to reply to this above. I’m not sure if I have strong feelings on the fragmentation. I will say that the likelihood of B/R moving to a different library is pretty slim. It’s already a lot of work to get all of our services converted to use vapor. I don’t think we’ll change the way we do configuration again just for kicks ;). That also means that I’ll be supporting Vapor for a while though.

I’ve developed Skogsrå a while ago, because we’ve found configurations could get messy in the long run when you have many releases. Right now, I’m using it extensively in:

• My personal project.
• Some of the open source libraries I maintain.
• The company I work for at the moment.

Though the startup I’ve developed this for doesn’t exist anymore, I kept using the library and adding the features I needed. The ones I use the most are the ones described here

Anyway, regarding the following:

I agree with @sasajuric. We (the maintainers of all those libraries) could join forces and actually develop an Elixir built-in solution. Our experiences configuring production systems all this years might come handy when developing an unified solution.

However, I also agree with this. This gives us a diverse ecosystem.

I still want to implement Dhall for Erlang so I would be able to use it with my Elixir projects. Other than that I just use sys.config or handle everything “manually” in Application.start/2 callback.

The way we approached this is by allowing the following spec:

{:db_name, dev: "my_db_dev", test: "my_db_test"}

This is interpreted as follows:

• in dev mode, the default is my_db_dev
• in test mode, the default is my_db_test
• both defaults can be overridden via os env
• there’s no default in prod - the app will fail to start if os env is not provided

I believe that this is a good trade-off between convenience and production safety. Dev defaults are a part of the codebase and we don’t need to keep .env files in the git repo. This reduced a recurring issue of a dev forgetting to update a .env file because the thing worked on their machine, and also removed a bit of duplication.

I’ve found confex pretty pleasant to use.

If that’s working for you then that seems fine. We’re pretty happy avoiding default values in code like that wherever possible and default to always loading from the same source regardless of the “mode” the app is in. We just try to avoid a distinction between “dev”, “test”, and “prod” wherever possible. We’ve found it less confusing when things go wrong and accommodates a wider range of developer workflows. But those are our preferences. They probably aren’t going to be correct for everyone.

I haven’t tried any tools yet (just maintaining releases.exs) - just waiting for the good patterns with built-in mix release

One thing I’d like to ask: what kind of validation besides type check (e.g. parsing to integer) do you want to put in config-read time not inside the app init?

I know there are some “bad config” we don’t want to even let the whole apps start… but doesn’t it make duplicate work? Shouldn’t we leverage apps and supervision trees instead?

Also I’m curious how these tools work well with Config.Provider - or whether we need some changes in Config.Provider.

Another idea: I’m wondering we can define common configuration spec in each app module, and expose back to config tools, not vice versa. By doing that, we can enforce the config is actually being passed (e.g. avoiding an error that I add config at config tool but not using it app…)