Rethinking app env

If a library Foo requires some parameters which you want to fetch from external sources you can invoke e.g. Foo.bar(fetch_params_from_external_source(...))?

I feel that’s not only good advice for library authors, but does also provide the opportunity to move the knowledge about the different ways of doing configuration (with it’s pros/cons) to users – instead of relying on library authors to choose a fitting way – and the developer can choose one way to do config instead of being forced into multiple ways of doing things by used libraries.

E.g. I really like the pattern of MyApp.Repo in the way it abstracts a set of configurations into a module I call, which could be generally useful for any library, as long as it’s not configured by it’s own app env preventing multiple parallel configurations.

How would you provide those parameters from regular code? I would think of something like I proposed here: Rethinking app env - #35 by StefanHoutzager (with the get_env(keys) call).

The approach I’m arguing for is to provide as much parameters as possible from the regular code. That code is runtime friendly so you can fetch most of your config from arbitrary sources, including external databases. You could even configure e.g. your endpoint by fetching the parameters from your repo.

How would you solve this problem

And how would you handle customer-specific application configuration (with software + db installed on a server per customer). Configuration should be possible for consultants, so a user-friendly way is let’s say extra preferred.

What I often saw as freelancer, working on huge administrative applications for customers, was very little config in files (not much more than db startup config), and customer-specific config in the database. Mostly the same database as where other applicationdata were maintained. The customer-specific config was configured by consultants. Application + db were on a separate server per customer.

I completely agree with everything you have said in this post!

Yes, as the title says, this thread is about app env and it’s closely related cousins config scripts.

Just to be clear, I don’t deny that there’s a thing called system configuration. I just think that app env and config scripts are in most cases an inferior mechanism for managing the system configuration. Moreover, I think that the defaults promoted by e.g. phx.new are arbitrary, confusing, and at the same time limiting.

In my impression, a bunch of stuff ends up in config scripts for dubious reasons, and they don’t even help with actual system configuration (assuming you want to run OTP releases). I’ve been personally confused by config scripts, the team I’m a part of has been confused by config scripts, I’ve seen other people being confused by them, and a lot of libraries promoting them for no particular reasons.

I think we need to strongly challenge this approach, and reevaluate do we really need to promote config scripts and app env so much. I don’t say that they are useless, but I think that in most cases they are not the most appropriate choice. I think that we should instead promote regular code as much as possible, and leave the special code for the few cases where it’s really needed.

1. It is a single point of failure for your application instances. 

You indeed have one single point of failure. But f.e.:

To see how generally difficult configuration management can be, we only need to
look at the simplest of configurations: a flag to tell our web servers whether we’re
under maintenance. If so, we shouldn’t make requests against the database, and
should instead return a simple “Sorry, we’re under maintenance; try again later” 
message to visitors. If the site isn’t under maintenance, all of the normal
web-serving behavior should happen.

In a typical situation, updating that single flag can force us to push updated configuration
files to all of our web servers, and may force us to reload configurations on all
of our servers, if not force us to restart our application servers themselves.

(https://redislabs.com/ebook/part-2-core-concepts/chapter-5-using-redis-for-application-support/5-4-service-discovery-and-configuration/5-4-1-using-redis-to-store-configuration-information/)

2 When should an application instance read from there? How does an application instance 
  know when its values are changed?

If a configuration (say the flag “web server under maintenance”) changes maybe the database (f.e. redis) can push the notification to the clients? I don’t know redis, maybe there are other db’s that can do that?
Edit: seems some db’s can do that: https://www.slant.co/topics/4270/~databases-to-push-updates-from-the-server-database-to-the-client-in-real-time

Actually, just because this happens to be configuration for you, that doesn’t mean it’s configuration for everyone. You see, given enough projects, there are all sorts of scenarios where various unlikely things end up being configuration (e.g. in our case, the database is configurable). So if we move from “it’s configurable to me” to “it’s therefore configurable for everyone”, we’ll just end up stuffing all the parameters into config scripts. No constant value would ever exist in the regular code. Good luck maintaining such code

The approach I’m arguing for in my article, and in other discussions, is to treat all of the parameters first and foremost as parameters, not some mythical configuration which has to be placed into some different place, distant from the code which uses it.

Then, you promote parameters into configuration when you actually have that need. Hence, if you want to vary the pubsub adapter, you explicitly make it configuration. With such approach your configuration grows organically, and consists of the stuff which is in fact configurable in your project.

The library does that by accepting parameters via functions or callbacks. It doesn’t have to do the guesswork and promote random things as configuration upfront. As a developer of your system, you’re way more familiar about its particularities than any library author. Hence the decision about what is configurable in your system should be left to you.

It’s a side-discussion, but since you mentioned it - if this is the only reason, I’m not sure it’s a good trade-off. You’re creating a distance between the actual version and the one you develop and test against, making it more likely to miss some issue until it hits the production. The mentioned benefit doesn’t seem very substantial. I haven’t worked with Redis in many years, but when I did, the installation was trivial, so somehow I doubt it’s complicated today. With a little bit of automation, e.g. using docker, you can simplify it even further.