What cannot be runtime config?

Questions / Help
1

Forgive me for maybe a noob question. I am reading much about configuration – maybe this in Elixir is one of the most confusing things? I read this Configuring your Elixir Application at Runtime with Vapor | AppSignal Blog and it says “if it can be set at runtime, it should be set at runtime.” This makes much sense to me – this is how other languages work with .env files – but maybe the comparison is not fair with dynamic languages?

My question is what things can not be runtime config? I imagine a very simple setup from working with PHP where there was only 1 config and 2 .env files (regular .env and a .test.env) and the test suite would know to use the .test.env. But dynamic languages do not have the same concerns. I am thinking maybe a config for a module name like saying which HTTP client the app should use. A system ENV maybe has string for “HTTPoison”, but you must convert this to atom like Elixir.HTTPoison. It is ugly, but I think this works fine. What things will not work this way and must be set in a config file for compile time config?

Thank you for your replies!

2 Likes
2

The first thing I can think off are modules names that macros will be used from, or will be modules generated by macros.

Basically, anything that changes the compilation result.

2 Likes
3

How to know what modules have macros?

4

Well if we are talking about your application then you already know it. And if we are talking about a library then it is either documented in the configuration section of the documentation, if it exists, or you have to dive in the code:

Module that define macros call defmacro. Module that use macros call use SomeModule or require SomeModule.

I am not sure if import SomeModule also “requires” the module to be able to call its macros.

Unfortunately this only gets you so far…

That is why, when they say " if it can be set at runtime, it should be set at runtime", then I would say “yes, if it is environment dependent (a hostname, a port, a database connection string, anything that you would put in your .env files)”. Otherwise, if it is something that doesn’t change, the compile-time config is fine.

5

I am wondering how to move most configs into runtime.exs. This works simple for local running or on production server when real values are required. runtime.exs looks something like

import Config

config :myapp,
  x: System.get_env("X"),
  y: System.get_env("Y"),
  z: System.get_env("Z"),
  # ... etc...

But how to override this for testing?

6

You can use Config.config_env/0 to differentiate between envs

import Config
... 
if config_env() == :test do
  config :myapp, foo: "bar"
end
7

You can use Application.put_env in your tests directly.

8

Forgive me for my struggle to understand a good solution. I think my options are to do maybe these:

Option 1: fill runtime.exs:

if config_env() == :test do
  config :myapp, 
      x: "test-x",
      y: "test-y",
      z: System.get_env("TEST_Z", "test-z"),
     # ... etc...
else
  config :myapp, 
      x: System.get_env("X"),
      y: System.get_env("Y"),
      z: System.get_env("Z"),
     # ... etc...
end

Pro: declarative, clear
Con: double long runtime.exs. Also if-statements in configs are always for me a smell.

Option 2: use test_helper.exs

Use simple runtime.exs and add test values to test_helper.exs like maybe this:

  config :myapp, 
      x: System.get_env("X"),
      y: System.get_env("Y"),
      z: System.get_env("Z"),
     # ... etc...

and then in test_helper.exs I can done Application.put_env like

Application.put_env(:myapp, :x, "test-x")
Application.put_env(:myapp, :y, "test-y")
Application.put_env(:myapp, :z, "test-z")

Pro: simple runtime.exs (no double, no if-statement)
Con: cannot enforce System ENV vars using System.fetch_env! – only get_env would work so that execution does not stop before test_helper.exs runs and a value is supplied.

Option 3: Vapor

I looked also at Vapor. I like this: it could replace runtime.exs 100% I think. But it is still a problem when you must manually take a value and use Application.put_env to put it into the correct place so your app and deps can find it. For example, imagine a dependency with config like this:

config :some_dep, 
   deep: %{
      thing1: "a",
      thing2: System.get_env("THING2")
   }

With vapor, I can get THING2 ENV var from .env but I must be careful with any deep config like this example dependancy. Where to put it once I have it? I think I must merge map and then put inside carefully into Application process dictionary.

Pro: support for .env and .env.{ENV} files for each override. Errors when config is not correct.
Con: difficult for complex config that rely on Application.get_env.

I think there must be other ways. Sorry for my confusion. I admit I thought this was a easy topic but I am understanding more that config has subtle!

1 Like
9

What about having in config/test.exs

  config :myapp, 
      x: "test-x",
      y: "test-y",
      z: System.get_env("TEST_Z", "test-z"),

and in config/runtime.exs

if config_env() == :prod do
  config :myapp, 
      x: System.get_env("X"),
      y: System.get_env("Y"),
      z: System.get_env("Z"),
     # ... etc...
end

[/quote]

if-statements in configs are always for me a smell.

For me if-statement is smell almost everywhere but configs :grinning_face_with_smiling_eyes:

Take a look at the example how configs are organized in hexpm/config at main · hexpm/hexpm · GitHub

10

This suffers from being long and containing ifs, but I’m posting it anyway for reference. I went 100% on runtime.exs in my project and only have a couple of things that need to be set at compile time in the config.exs.

This has worked well for me. Even though the runtime config file is big, it’s not really that big once you get to know it, and I can have unified config for both development and release time (just using .env in development for convenience).

7 Likes
11

This is a beautiful little package! Thank you for sharing your approach – it works well!

1 Like
12

Wow this really made me think. Thank you for this strategy! I tried it and it works very well – the hex suggestion was helpful for study.

13

There are basic 2 types of config in Elixir code:

  1. Compile time configuration. Before Elixir 1.10 you could call Application.get_env/{2,3} in any part of the code, even compile time, so people often did stuff like:
    defmodule Foo do
  @option Application.get_env(:my_app, :foo_config)

  # …
end
    This caused the @option to be set during compilation, so if someone was using releases (Distillery or Relx at the time) could be confused why @option didn’t changed when the configuration was changed on the target server. Since 1.10 we have Application.compile_env/{2,3} and Application.get_env/{2,3} will print warning when used in compile time. There are few examples of applications that use compile time env and in few situations this still sometimes makes some sense (for example poor-man dependency injection).
  2. Runtime environment, that is Application.get_env/{2,3} calls within functions body, that is it. The “funny” thing there is that runtime configuration is also divided in 2 separate things (and people often forgot about it):
    1. “Startup config” that is read only at the beginning of application lifetime (often in module that is implementing Application behaviour, or in init/1 functions of the servers). This is for example reason why simple Application.put_env(:logger, :level, :warning) will not work, and you need to use Logger.configure(level: :warning) or reason why you cannot configure :kernel and :stdlib applications via config/config.exs (so for example configuration of Erlang’s logger is currently enormous PITA in Elixir).
    2. “On-demand” configuration where Application.get_env/{2,3} is called “in-place”, where it is needed. Reason why a lot components cannot be configured that way are mainly 2:
      • Performance - reading environment is much slower than just variable from the process state
      • Data is needed at the start of the process - for example you cannot change port on which Cowboy is listening for connections after it was started (obviously)

So 1. is obviously something that you cannot configure in runtime (examples of such behaviour is :mime database). And depending when your Vapor configuration will be ran 1.1. can also be off limits, at least for some applications (notably :kernel and :stdlib are hard to configure with any Elixir tooling).

5 Likes
14

Interesting, thank you! I am wondering can you clarify somethings?

Performance - reading environment is much slower than just variable from the process state

Is Application.get_env/3 reading from the process state?

How is configuration of Erlang’s logger PITA? (I am only familiar with basic log level setting)

15

It is more like reading from ETS AFAIK. That is why it is not the best performance-wise when used in environment that need to be as fast as possible (like logger calls).

You cannot do stuff like:

config :kernel, :logger, [
  # …
]

In your config/config.exs nor in config/runtime.exs as when these files are evaluated by Mix kernel application is already started and running. This mean that you cannot easily configure logger in development that will work from the beginning of the VM lifetime (like for example SASL startup messages).

Let just say, that this isn’t very popular approach in Elixir development to use Erlang’s logger to full extent, but with Elixir 1.11 it can became more and more feasible to do so, as Elixir backends do not have simple access to structured logging (it will always be translated to plain string) nor to all log levels (these have access to “basics” - debug/info/warn/error, while Elixir 1.11 supports all 7 syslog levels). There are PRs to OTP to improve that, but it will take some time before that functionality will be available in Elixir.

2 Likes
16

The Application.env is 100% ets:

iex(serenity@wintermute.skunkwerks.at)9> :ets.tab2list :ac_tab
[
  {{:application_master, :credentials_obfuscation}, #PID<0.367.0>},
  {{:env, :gettext, :default_locale}, "en"},
  {{:env, :kernel, :logger},
   [
     {:handler, :default, :logger_std_h,
      %{
        config: %{type: :standard_io},
        formatter: {:logger_formatter,
         %{legacy_header: true, single_line: false}}
      }}
   ]},
  {{:application_master, :cowboy}, #PID<0.415.0>},
  {{:env, :amqp_client, :writer_gc_threshold}, 1000000000},
  {{:env, :lager, :crash_log_count}, 5},
  {{:env, :ex_unit, :timeout}, 60000},
  {{:application_master, :goldrush}, #PID<0.326.0>},
  {{:loaded, :logger},
...

You can look in the observer app, if you select view -> ETS tables -> system tables too.

For better (read-only) performance, look at Erlang -- persistent_term the persistent term cache, added in OTP21.2. Note that updating this is a java-esque stop-the-world scenario for the entire VM.

4 Likes