What is the difference between using {:system, "PORT"} and System.get_env("PORT") in deployment?

[OvermindDL1]

@josevalim

I have a hunch you are confusing the inits

Probably, I still have the flu and most of this thread was a blur to me due to lack of being able to concentrate. ^.^

[josevalim]

@sasajuric

You could consider decoupling the callback module from the behaviour module. I actually dislike the fact that the Endpoint conflates the plug module and the process behaviour.

This is unfortunately trickier than expected because both rely on compile and runtime configuration. So it could work but it also means you will end-up configuring two modules in the application environment, let’s say MyApp.Endpoint and MyApp.Pipeline.

One option we discussed was to use init/2, where the first argument is the environment and the second are the options:

def init(:prod, opts) do
  port = System.get_env("PORT") || raise "expected the PORT environment variable to be set"
  {:ok, Keyword.put(opts, :http, [:inet6, port: port])}
end
def init(_env, opts) do
  {:ok, opts}
end

Would you consider the approach above superior to the one currently in master?

[OvermindDL1]

@josevalim

Would you consider the approach above superior to the one currently in master?

I’m not really a fan of passing the environment with init, nor of accessing System.get_env either, very hard-coded. I’d really like for some Config module or so to handle all this, basically like:

def init(opts) do
  opts = Config.update_if_not_exist(opts, :port, 42)
end

Where Config.update_if_not_exist could works with Keyword’s or maps, if the looked for key already exists then it uses it, else it looks in the current Application environment (might be good to pass that in explicitly), else it takes a default. If any of the existing value, looked up value from the application environment, or the default is in the form something like %SystemEnv{key: "PORT"} (perhaps with an optional default: that defaults to nil) then it looks it up in the system environment. The nice part of this is that it could be a protocol so someone could make their own lookup that could look up in a database or a remote server or whatever.

However, a nice thing about such a Config call is that the environment could be baked into it (an attribute with a getter?), and it could do something like look up the application environment key of the environment, say :prod, if it exists then use any options configured on it, if an option is missing then check the option on the main application environment, if it is missing then use the default that is passed in.

Such a thing would still be backwards compatible with the current config infrastructure, allows easy migration to the new formats and allows you to plug in external run-time lookups as well?

This may be out of scope of what is being discussed here, I’m still woozy, but I’ve always wanted a configuration like this for erlang where you can specify an option in the opts, or the application environment, or some external config file, or some arbitrary-user-defined run-time lookup with whatever is being used able to be defined by the user, not the module author. ^.^

/me goes to take more sudafed…

[josevalim]

It is not Phoenix goal to provide a complete solution for configuration management. The goal with the snippet above is to provide an example of where such configuration lives. If someone wants to use A, B, C or Z, choose whatever you want as you think it suits your needs.

[talentdeficit]

i think presenting init as a solution to the problem of resolving appups is misleading. it’s already the case that you have to manually mark supervisors and any modules that are called during init as changed to get proper restarts of processes. saying 'just do any dynamic config in init’ doesn’t change that

[sasajuric]

@josevalim

And again, reading from the environment doesn’t impact dynamic configuration because the unique keys are based on the module name and you always need at least a module name.

The thing I’m aiming at with dynamic endpoints is the case where I have just one endpoint plug, and start multiple endpoints dynamically using that plug, but with different configurations. So module name as key won’t work here. I’ll have to do some MyEndpoint.1 with Module.concat or other similar ugly tricks. Compile-time settings also get in the way here, so the less we have them the better.

@josevalim

Please provide an example where not reading from Application.get_env actually makes it possible to do something that is not possible in master.

With the current master it seems everything is possible, but definitely less than perfect.

@josevalim

Please describe a system that can take into account multiple environment and compile-time configuration. Assuming compile-time options do not exist is not a viable option for Phoenix nor Ecto.

I’m less interested in compile-time options b/c those are not flexible anyway, meaning I need to derive their values during compilation. So I’m fine with passing those through app env, though it would be nicer to be able to explicitly pass them as args to macros (use and others).

The fact that such args would be treated differently than run-time args has a nice benefit that it’s clear which options can in fact be provided through the callback (or through start_link args).

@josevalim

That’s exactly why I said you can solve each independently, but please try to consider a solution that solves all of them.

Given that these options are applied at different phases, why do we need one way to solve all of them?

@josevalim

One option we discussed was to use init/2, where the first argument is the environment and the second are the options:

Assuming that init/2 is then always invoked (so no magical config setting), I’d find it better than the current proposal.

An alternative could be something like:

if Mix.env == :prod do
  def init(opts), do:
else
  def init(opts), do:
end

Since init/1 is taken some other name should be chosen for the callback function.

[josevalim]

@talentdeficit

i think presenting init as a solution to the problem of resolving appups is misleading.

Indeed. It is not a solution but a requirement for smoother appup-ing.

@sasajuric

The thing I’m aiming at with dynamic endpoints is the case where I have just one endpoint plug, and start multiple endpoints dynamically using that plug, but with different configurations.

You won’t need multiple names because the configuration is not read exclusively from the environment. You can update it dynamically on init (and also pass it during start_link in Ecto’s case).

@sasajuric

The fact that such args would be treated differently than run-time args has a nice benefit that it’s clear which options can in fact be provided through the callback (or through start_link args).

That would be a pro but there is a huge con in that it would be impossible for us to make a compile time configuration become a runtime configuration without breaking the user code. For example, I converted three or four configurations in Phoenix from compile to runtime in the past week. If they were in different places, I would have broken everyone’s application.

@sasajuric

Assuming that init/2 is then always invoked (so no magical config setting), I’d find it better than the current proposal.

Thank you for the feedback. I don’t like the Mix.env option, because while we are only using it at compile-time, I am afraid folks would not be clear with the distinction and start using Mix.env in their code, which is not recommended (as Mix.env is not available in releases).

[michalmuskala]

I think having a fixed function will be much better than the current one in config. As to the name for init/2 we could pass :supervisor as the first argument to mirror ecto - otherwise we could use supervisor_init/1. I agree that passing the environment as argument is a bit wired.

[josevalim]

That’s a good point but I had this exact same discussion with Chris. If we don’t pass the environment, how we are going to choose when to load values from the System.get_env? It likely means a value needs to be passed through start_link or an option need to be set on the config which will be checked in the callback, which leads to be convoluted code. Something like:

def init(_kind, opts) do
  if opts[:load_from_system_env] do
    port = System.get_env("PORT") || raise "expected the PORT environment variable to be set"
    {:ok, Keyword.put(opts, :http, [:inet6, port: port])}
  else
    {:ok, opts}
  end
end

where in your config/prod.exs you will set:

config :my_app, MyApp.Endpoint, load_from_system_env: true

Is this better or worse than what we have today?

[sasajuric]

@josevalim

That would be a pro but there is a huge con in that it would be impossible for us to make a compile time configuration become a runtime configuration without breaking the user code. For example, I converted three or four configurations in Phoenix from compile to runtime in the past week. If they were in different places, I would have broken everyone’s application.

You could deprecate the compile time property.

@josevalim

I don’t like the Mix.env option, because while we are only using it at compile-time, I am afraid folks would not be clear with the distinction and start using Mix.env in their code, which is not recommended (as Mix.env is not available in releases).

Since many people enter Elixir through Phoenix, and they all touch configuration code pretty early, I believe this is in fact a great opportunity to teach them about Mix.env and explain when it shouldn’t be used by including one or two sentences in the generated code.

@michalmuskala

As to the name for init/2 we could pass :supervisor as the first argument to mirror ecto - otherwise we could use supervisor_init/1

We could call it dynamic_configuration or something of the sort, because that’s really why we have it there, right? Although I guess an added bonus is that one can do additional initialization of the endpoint.

[michalmuskala]

You could set the port from system env, whenever it’s not set from app env:

def init(_kind, opts) do
  if opts[:http][:port] do
    {:ok, opts}
  else
    port = System.get_env("PORT") || raise "expected the PORT environment variable to be set"
    {:ok, Keyword.update(opts, :http, &[{:port, port} | &1])
  end
end

[josevalim]

Thanks, I will discuss all of those options with Chris and followup.

[Virviil]

I’m a bit confusing about configuring packages that are not startable - like simple package of functions.

They don’t have any supervision tree, no modules that can be started - but they also can be configurable.

So, how to configure this types of “applications”?

I prefer to use “:system tuple” configuration approach, but after reading this topic i’m not sure that this is the best idea…

[michalmuskala]

I would say that those packages being configured through application env, is simply misusing application env. If a package is a bunch of pure functions, it should accept configuration through arguments to those functions.

[Virviil]

Let’s for example imagine API wrapper. You want to encapsulate http requests and all other like error decorating. Thus for example I want to call MyRestEndpoin.new_user("Alice") but not MyRestEndpoin.new_user("Alice", hackney: [basic_auth: ...], proxy: ..., app_token: ... secret: ...).

I think that you can find what to move to config section from the library in any cases.

Or you suggest to make all such libraries as applications? May be this is really good idea…

[kaelumania]

@josevalim @sasajuric I really like the explicity and simplicity of handing in config_parameters or better all dependencies explicitly via a constructor like start_link instead of having a hidden mechanism.

[PavelTyk]

Have you came up with a solution?

[ardhitama]

Reviving this old thread since I stumbled upon a system which mixed Syste.get_env and :system tuple.

import Config

config :app,
  an_url:  {:system, "AN_URL"},
  url_secret: System.get_env("URL_SECRET"),

Not sure but somehow the an_url can be accessed by
Application.get_env(:app, :an_url)
but url_secret can’t with
Application.get_env(:app, :url_secret)

Most probably it’s documented somewhere but my google fu is not there yet.
Any explanation of this? or I’m being a dumbo

[D4no0]

This article should cover exactly in detail on why your System.get_env("URL_SECRET") doesn’t work as you are expecting.

Think of it as the following:

• {:system, "AN_URL"} - A config declaration that will be read at runtime, this means on the deployed system, this is deprecated now in favor of runtime configuration;
• System.get_env("URL_SECRET") - If this is declared in any config files, the exception is runtime.exs, the environment variable will be read and populated at compile-time, aka when the project is built.

[ardhitama]

that makes sense! though to my defense the :system tuple is not mentioned in the doc