Rethinking app env

So then all the stuff in config.exs is not configuration at all? Also, all the things which are the same in dev/test/prod.exs are also not configuration? Finally, the things which are fetched from external sources (e.g. at Aircloak we fetch database parameters from a json file) are also not configuration?

This also confuses me. Config scripts are a part of the codebase.

1 Like

Indeed, I think that many of those are not really configuration at all, unless you think that it might be changed for all or some of your app’s instances in the near future (which could be called different instances of the same app). What I mean with (not) ‘in your cosebase’ is that not all configuration, especially not your production configuration, is part of your source code repository (you do not copy them over using git or similar tools). Also, configuration files always lives at the peripheral of your system, i.e. outside of the ‘lib’ folder for Elixir systems.

Let me elaborate a little more, since I finally am back on the computer rather than on my phone:

Files like config.exs contain configuration that might be applicable to all your app instances today, but:

  1. It provides default values that are overridden by more specific configuration settings in other configuration files. How much of this is applicable vs. how much of this should just happen in code is subjective; writing the default values here is more explicit (so it is easier to find the default value), but if adding a specific configuration setting is the only way to make a dependency library work, then I’d say the dependency is more leaky than it should be.
  2. It provides values that might differ between the app today and the app tomorrow (i.e. the near future). Examples of these might be the logger level or network connection strategies of a Nerves system, the amount of encryption rounds to use for your PKBDF2 (which you want to increase over time as hardware becomes better) etc.
  3. When there is no sensible default because the library does not care (the chosen value depends on the environment of the application (the (type of) system the app runs on) rather than the implementation of the application, but a choice has to be made. Settings like the Erlang Cookie, the Phoenix Endpoint Secret Key Base, the folder into which attachment files should be uploaded, etc. fall into this category.

There is obviously some overlap between (1), (2) and (3) in certain cases.

In this way, what is inside your config.exs is configuration. And what is in your dev/test/prod.exs is configuration. And the database settings you receive from an external location are definitely configuration because you probably are not doing this in the development- or testing-instances of your application.


An example of something that I think is ill-suited for specifying in configuration, and rather should be part of your application’s real code itself, would for instance be what locales you’d like to use for CLDR. This is a very interesting project, but the way you specify what locales your app requires is by adding this to your configuration. They even include a special ‘compiler’ so changes to its configuration are picked up as ‘code changes’. For the locales, I think it would make more sense if inside a module that would like to use CLDR, use CLDR, locales: [:en, :nl, :de] would be used, with the implementation of CLDR’s __using__-macro gathering the required locales in that way.

Another problem over the high amount of settings that libraries shift off to configuration files, is that configuration is effectively global for your application, so using the library in one ‘configuration’ for one part of your OS-app (like one of your BEAM apps) and a different one for another becomes impossible.

There are some other libraries I’ve used in the past that just don’t do anything, or even throw errors at app launch unless you’ve added certain snippets to your configuration, although I am currently not able to remember them (I also believe I did not end up using them exactly because this).


I like libraries that allow an (OS-)app-wide global setting with module-local overrides (by e.g. using the use Foo, config: ... syntax or similar). I also like what Decimal does with the Decimal.with_context(fn ->... end) construct to (temporarily) alter rounding behaviour.


I hope this gives some more context :slight_smile: .

2 Likes

So then this stuff, injected by phx.new doesn’t belong here at all?

config :my_site, MySiteWeb.Endpoint,
  # ...
  render_errors: [view: MySiteWeb.ErrorView, accepts: ~w(html json)],
  pubsub: [name: MySite.PubSub, adapter: Phoenix.PubSub.PG2]

How exactly is it easier to find the default value in a config script? I need to first know that the value is provided by config scripts (since in fact many values are not defined in config scripts), and I also need to know its name. So it looks like I still need to consult the code first, possibly also read a library documentation, before I can find the value.

How do I determine such values? The examples you’ve mentioned are the things which IME change maybe once or twice in a period of few years.

I’ve just checked the git log of one of our project’s config folder. In the past year we had some changes there. Most of them were additions of new properties, some of them were deletions, and I was able to find only one case where some value has actually been changed. And that change was due to a Phoenix upgrade, not due to “reconfiguration”. Our config scripts were more frequently modified due to making comments, than due to changing a value of some “configuration”.

We actually had more frequent changes of values typically provided in the regular code (e.g. Supervisor parameters, GenServer timeouts) than “config” values.

Perhaps we completely failed in organizing our configuration?

I’m confused here. The examples you mentioned don’t end up in app env (Erlang cookie is a VM arg, not provided by a config script), or don’t need to be in app env (secret key base).

I’m not much wiser :slight_smile: The only clear rules I can make so far are:

  1. Use config script if required by library
  2. Use config script if values change between different mix env

The first reason can’t be avoided (although opening up an issue with the library might help).

The second reason deserves to be questioned. Given that Elixir has other means of making a decision based on mix env, why are config scripts the best approach?

@Qqwy your point on cldr is very well taken and for Version 2 that is exactly the approach I am taking. Its much cleaner - but it took me a lot of learning to figure that out.

I am still a little uncomfortable with the complexity associated with creating user-defined module(s) to host the compile-time generated functions when the public API generated has a large surface areas as cldr does - but thats for another topic.

2 Likes

Some quick thoughts. A lot of config would be easier to maintain when it resides a database, via a gui. Access can be authorised, you could version the records. Your businesslogic should have no knowledge where the config resides, so you could get the config via a call get_config(keys) to a separate application (the data access layer) that knows ecto (and config files). In the dataaccess layer is decided from which store to get the config. Env type, version etc can be taken into account there. Not thought about it yet, but maybe a rules engine could help making things flexible also http://www.myti.it/blog/2015/5/12/how-to-develop-a-product-configurator-software-using-a-rule-engine, it is not difficult to build the backend in elixir. I built one for one of the rulestypes and use the opensource frontend from Camunda (a DMN editor).

2 Likes

I indeed think it would make more sense if this were to be part of the MySiteWeb.Endpoint module.

Yes, you are right. It probably is a very weak argument. You really do need to read through the configuration file to find out what happens, and have to be able to understand how the data types that are written there correspond with actual behaviour, so digging through documentation is required. So you are right, this is not a good reason.

Well, logging is something I change more often (temporarily changing it to a lower log level and later back) to check in more detail when something goes wrong. There is no ‘failure’ here because the world is not that black and white. Network connection strategies are something that change based on the physical location where a Nerves device will be used. PKBDF2 is something that has not been changed so far since our application has not been running that long in production, but at some point it will.
I think, rather than creating a very clear definition of near (which would be very subjective and not that useful), it is probably more important to think about cohesion/coupling of your settings w.r.t. the implementation inside the application: If there are multiple related behaviours your app-part (like BEAM-app or library) might provide, then putting these on the outside of the library rather than hard-coding them means that the app-part is more flexible in a wide range of environments.

Ahaha! I think we are getting somewhere here! It seems you are talking specifically about configuration script *.exs files, while I am talking about the concept of configuration in general (including vm.args scripts, loading settings from system environment variables etc.). Maybe that explains some of the confusion.

This indeed is a difficult question, and I fully agree with you that we should question this, since config scripts are by no means to be considered ‘holy’. I currently am of the opinion that they are not necessarily bad, but over-used (and that configuration in general might be over-used in some Elixir-apps, which was the reason behind the couple of posts I wrote so far).
I definitely agree that looking into different ways to make environment-based choices is a good idea.

So maybe better guidelines (let’s not call them ‘rules’ because it is highly dangerous to claim universal applicability):

  1. Use config scripts if required by library (which we might try to avoid by releasing a new version of the library that does not depend on this).
  2. Put values that change between individual app instances inside the configuration (whether in config scripts or somewhere else, but I think this is a prime candidate to do put inside a config script). Examples: node name, server name, local database connection info.
  3. Put values that change between environments somewhere in configuration, but not necessarily inside your config scripts.
  4. Put values that are likely to change somewhere in your configuration (rather than hard-coding them in a way that might result in strong coupling), but not necessarily in your config scripts.
2 Likes

Wow I’m pretty new to elixir and I have been following the conversation around this intently.
It’s really great to hear from more the more experienced devs on how they approach this.

For what it’s worth I also find it very confusing now to decide what should go in config files.

I have come across situations specifically with plugs, where some configuration get’s passed to init/1 at compile time, and other config that gets pushed into config files. For example:

plug :authorization, strategy: :token

...

config :authorization, token_lifetime: 3600

I see this split as the :token_lifetime being something that might be changed at runtime in the app env, or different in a different env like test. But it still means you are looking in 2 different places for the config or your authorization.

Maybe one size will never fit all. But it would be great to see more best practises/thoughts.

1 Like

While I think that configuration databases might be applicable for some applications, there are a couple of important drawbacks:

  1. It is a single point of failure for your application instances.
  2. When should an application instance read from there? How does an application instance know when its values are changed?
2 Likes

In my case, this is actually something that varies between environments. I use redis for prod, but PG2 for dev and test so that the redis dependency isn’t necessary. It’s important to consider that even if your particular usage of a libraries configuration does not vary, someone else’s might. I think it’s important for a library to consider the more general case of how configuration may vary across everyone’s use cases.

1 Like

Actually, just because this happens to be configuration for you, that doesn’t mean it’s configuration for everyone. You see, given enough projects, there are all sorts of scenarios where various unlikely things end up being configuration (e.g. in our case, the database is configurable). So if we move from “it’s configurable to me” to “it’s therefore configurable for everyone”, we’ll just end up stuffing all the parameters into config scripts. No constant value would ever exist in the regular code. Good luck maintaining such code :slight_smile:

The approach I’m arguing for in my article, and in other discussions, is to treat all of the parameters first and foremost as parameters, not some mythical configuration which has to be placed into some different place, distant from the code which uses it.

Then, you promote parameters into configuration when you actually have that need. Hence, if you want to vary the pubsub adapter, you explicitly make it configuration. With such approach your configuration grows organically, and consists of the stuff which is in fact configurable in your project.

The library does that by accepting parameters via functions or callbacks. It doesn’t have to do the guesswork and promote random things as configuration upfront. As a developer of your system, you’re way more familiar about its particularities than any library author. Hence the decision about what is configurable in your system should be left to you.

It’s a side-discussion, but since you mentioned it - if this is the only reason, I’m not sure it’s a good trade-off. You’re creating a distance between the actual version and the one you develop and test against, making it more likely to miss some issue until it hits the production. The mentioned benefit doesn’t seem very substantial. I haven’t worked with Redis in many years, but when I did, the installation was trivial, so somehow I doubt it’s complicated today. With a little bit of automation, e.g. using docker, you can simplify it even further.

3 Likes
1. It is a single point of failure for your application instances. 

You indeed have one single point of failure. But f.e.:

To see how generally difficult configuration management can be, we only need to
look at the simplest of configurations: a flag to tell our web servers whether we’re
under maintenance. If so, we shouldn’t make requests against the database, and
should instead return a simple “Sorry, we’re under maintenance; try again later” 
message to visitors. If the site isn’t under maintenance, all of the normal
web-serving behavior should happen.

In a typical situation, updating that single flag can force us to push updated configuration
files to all of our web servers, and may force us to reload configurations on all
of our servers, if not force us to restart our application servers themselves.

(https://redislabs.com/ebook/part-2-core-concepts/chapter-5-using-redis-for-application-support/5-4-service-discovery-and-configuration/5-4-1-using-redis-to-store-configuration-information/)

2 When should an application instance read from there? How does an application instance 
  know when its values are changed?

If a configuration (say the flag “web server under maintenance”) changes maybe the database (f.e. redis) can push the notification to the clients? I don’t know redis, maybe there are other db’s that can do that?
Edit: seems some db’s can do that: https://www.slant.co/topics/4270/~databases-to-push-updates-from-the-server-database-to-the-client-in-real-time

1 Like

Yes, as the title says, this thread is about app env and it’s closely related cousins config scripts.

Just to be clear, I don’t deny that there’s a thing called system configuration. I just think that app env and config scripts are in most cases an inferior mechanism for managing the system configuration. Moreover, I think that the defaults promoted by e.g. phx.new are arbitrary, confusing, and at the same time limiting.

In my impression, a bunch of stuff ends up in config scripts for dubious reasons, and they don’t even help with actual system configuration (assuming you want to run OTP releases). I’ve been personally confused by config scripts, the team I’m a part of has been confused by config scripts, I’ve seen other people being confused by them, and a lot of libraries promoting them for no particular reasons.

I think we need to strongly challenge this approach, and reevaluate do we really need to promote config scripts and app env so much. I don’t say that they are useless, but I think that in most cases they are not the most appropriate choice. I think that we should instead promote regular code as much as possible, and leave the special code for the few cases where it’s really needed.

7 Likes

I completely agree with everything you have said in this post! :+1:

2 Likes

What I often saw as freelancer, working on huge administrative applications for customers, was very little config in files (not much more than db startup config), and customer-specific config in the database. Mostly the same database as where other applicationdata were maintained. The customer-specific config was configured by consultants. Application + db were on a separate server per customer.

How would you solve this problem

And how would you handle customer-specific application configuration (with software + db installed on a server per customer). Configuration should be possible for consultants, so a user-friendly way is let’s say extra preferred.

The approach I’m arguing for is to provide as much parameters as possible from the regular code. That code is runtime friendly so you can fetch most of your config from arbitrary sources, including external databases. You could even configure e.g. your endpoint by fetching the parameters from your repo.

1 Like

How would you provide those parameters from regular code? I would think of something like I proposed here: Rethinking app env (with the get_env(keys) call).

I feel that’s not only good advice for library authors, but does also provide the opportunity to move the knowledge about the different ways of doing configuration (with it’s pros/cons) to users – instead of relying on library authors to choose a fitting way – and the developer can choose one way to do config instead of being forced into multiple ways of doing things by used libraries.

E.g. I really like the pattern of MyApp.Repo in the way it abstracts a set of configurations into a module I call, which could be generally useful for any library, as long as it’s not configured by it’s own app env preventing multiple parallel configurations.

2 Likes

If a library Foo requires some parameters which you want to fetch from external sources you can invoke e.g. Foo.bar(fetch_params_from_external_source(...))?

1 Like