I notice that config/*.exs have
http: [port: {:system, "PORT"}]
but I know that I can do this ,too
System.get_env("PORT"). What is the benefit of first one and the second one.
7 Likes
If you made your config
http: [port: System.get_env("PORT")]
then you would need to provide the PORT environment variable at build time which is when that code would be executed.
{:system, "PORT"}
is not some magic code to retrieve an environment variable, it is a setting that tells Phoenix to retrieve the port number from the PORT environment variable at runtime. This can be seen in the code at https://github.com/phoenixframework/phoenix/blob/996a83a27d8ccdc7e0e3bdda9c21d537b19b2002/lib/phoenix/endpoint/server.ex#L41
13 Likes
There’s one more thing worth adding: this is not something that happens automatically. I.e. you can not safely assume that the library you use will understand runtime configuration the same way Phoenix does.
I found a number of examples where this worked, but also happened that I was banging my head against the wall trying to figure out why something does not work the way I expected - only to learn that the lib I used does not understand {:system, "SOMETHING"} env config pattern.
11 Likes
This is one of the reason I avoid using {:system, ...} approach. Another is that when I need to determine some setting at runtime, the source is not always OS env.
So instead, in the app’s start callback before starting the supervision tree I fetch additional config params (e.g. from OS env, a file config, or external k-v service), and merge them into app env. This feels a bit hacky, but at least it’s straightforward and gives me a lot of options.
I’m somewhat annoyed when library authors force their users to provide parameters through app env. It makes it harder to configure settings at runtime. For example, I’d be happier if I could start Phoenix endpoint using something like MyEndpoint.start_link(config_params). Configuring through config.exs is then optional, but still as easy as:
MyEndpoint.start_link(Application.fetch_env!(:my_app, MyEndpoint))
Phoenix project generator could still generate the code which relies on app env, but I’d have a way to opt out.
It would also be nice if Mix.Config.merge was available in Elixir app, perhaps as Keyword.deep_merge. That way we could have some parts of configuration in config.exs, and fetch other settings at runtime:
Application.fetch_env!(:my_app, MyEndpoint)
|> Keyword.deep_merge(runtime_settings())
|> MyEndpoint.start_link()
I’ll ping @josevalim and @chrismccord, because I’m curious about their thoughts on this.
7 Likes
Yup.
I think the start_link configuration is also supported by both Ecto and Phoenix. The trouble with using it though is that some of those configurations are dependent on the environment or are compile-time based. For example, I may want to set the PORT only in production and for development the default of 4000 is fine. Doing it when calling start_link means pushing environment concerns to the caller of start_link.
In any case, the {:system, _} approach is a work around and I am not sure of a better solution. One of the ideas I had in mind is to support lazy application configuration. The idea is that we would have a struct called %Mix.Config.Lazy{module: ..., function: ..., args: ...}.
When loading the Mix configuration, we would automatically expand those the lazy structs by calling module, function, args. For example, environment variables would be stored as %Mix.Config.Lazy{module: System, function: :get_env, args: ["VAR"]}. Mix.Config would also support a mode to read those values WITHOUT expanding the lazy constructs, so they could be extracted by release tools and converted into proper VM start-up configuration.
5 Likes
Don’t know about Ecto, but I’m pretty certain that it’s not supported in Phoenix. Only start_link/0 is exported, and looking at the code it seems to me that configuring the endpoint is mandatory. Even tests modify app env before starting the endpoint.
Oh, that could be handled with something like:
Application.fetch_env!(:my_app, MyEndpoint)
|> Keyword.deep_merge(runtime_settings(unquote(Mix.env)))
|> MyEndpoint.start_link()
defp runtime_settings(:prod), do: [http: [port: ...]]
defp runtime_settings(_), do: []
I occasionally use this technique and have no bad conscience about it 
Having Phoenix (and other libs) accept settings as function argument would suffice for me 
Lazy approach looks cool. Perhaps some macro might make it a bit more DSL-ish.
We could support it in Phoenix but we do need a big disclaimer it is only for some configurations.
I occasionally use this technique and have no bad conscience about it
But you know quite well the consequences of using Mix.env, given you did an unquote. Using Mix.env inside lib is something we typically discourage because it is easy to get it wrong and add a runtime dependency on Mix. Maybe we could export the environment in the endpoint at compilation time?
Lazy approach looks cool. Perhaps some macro might make it a bit more DSL-ish.
Right. We would need something on top but the idea behind the scenes is the one I posted. Mix.Config could include a system(…) function for example that returns the lazy thingy.
Not sure what you mean, and I think there’s some misunderstanding here. The “only” changes I’d like to see are:
- Add
start_link/1to endpoint which accepts settings (keyword) as argument. - Change the generator to use
start_link/1and explicitly load app env config for the endpoint - Consider soft-deprecating
Endpoint.start_link/0 - Consider adding
Keyword.deep_mergeto Elixir to assist merging configurations.
Strictly speaking, point 1 is IMO the most important, though others would be nice to have as well.
As I said, I don’t think a lib should enforce using app env (which is what Phoenix does, and I’ve seen it in other libs as well). All points above would mean that Phoenix would move to “configuration through function arg” approach, but still use app env to provide the configuration. However, usage of app env is now pushed to the user’s code, so they have total control and can tweak it however they want, as I did in the sketch above.
Adding some comments in the generated code which explain how runtime configuration can be used would also help. In fact, Phoenix generator could even generate the code which uses config port for dev/test, and OS env for prod, similar to the sketch in my previous post. Then, the {:system, os_env} hack might also be soft-deprecated.
Assuming such changes are approved, I won’t mind providing the PR
3 Likes
No misunderstanding. What I meant is that it is fine to support start_link/1 for customized configuration like yours. However, I wouldn’t recommend your pattern as a general thing because it relies on unquote(Mix.env). It is totally fine for you to do it, and we should support it, I just don’t think it is a general solution.
OK, I see now what you mean.
Personally I think it’s a straightforward solution to the problem of “I want to do one thing in dev and another in prod”, where the difference is not just in the value, but also in the way the value is produced. One could also conditionally define a function with:
case Mix.env do
:dev ->
defp runtime_settings(), do: []
:prod ->
defp runtime_settings(), do: [...]
...
end
I personally think both approaches are fine and straightforward solutions to a not completely trivial problem.
Exposing the build env in the endpoint is an interesting idea, with the caveat that it solves the problem only for Phoenix. But I guess it’s still better than the current {:system, os_env} hack.
Alternatively, could some macro or special form be introduced, such that when I write say __MIX_ENV__ or __BUILD_ENV__ I get the env?
I’ve personally always wanted the config arguments to be able to take either a value, as they do now, or an anonymous function, that could be called when the value is accessed; right now the user of the environment has to test if it is a function and call it, which is very much like the {:SYSTEM, blah...} hack, but more generally useful.
1 Like
I want to update this old discussion and thread to say we have come up with a mechanism that no longer requires {:system, ...} and it is slowly being deprecated from Phoenix, Plug and Ecto.
James Fish pointed out the best place to do configuration is inside init/1. Doing it before calling start_link is an issue because then it won’t work with hot code reloading. Therefore with Phoenix v1.3, we are going to push dynamic configuration, such as the ones loaded from the environment, to the init functions or on_init hooks.
For example, here is how config/prod.exs looks like:
config :my_app, MyApp.Endpoint,
http: [:inet6, {:system, "PORT"}]
And here is how it looks on Phoenix v1.3:
# For production, we often load configuration from external
# sources, such as your system environment. For this reason,
# you won't find the :http configuration below, but set inside
# MyApp.Endpoint.load_from_system_env/1 dynamically.
# Any dynamic configuration should be moved to such function.
config :my_app, MyApp.Endpoint,
on_init: {Demo.Web.Endpoint, :load_from_system_env, []},
Where load_from_system_env is defined in the endpoint as:
@doc """
Dynamically loads configuration from the system environment
on startup.
"""
def load_from_system_env(opts) do
port = System.get_env("PORT") || raise "expected the PORT environment variable to be set"
{:ok, Keyword.put(opts, :http, [:inet6, port: port])}
end
This should answer all questions folks have with dynamic configuration, it doesn’t matter if it comes from System.get_env, the filesystem, etc.
Unfortunately we could not use init/1 for Phoenix, because the endpoint is already a Plug that requires an init function, and because the Phoenix Endpoint is an umbrella of multiple supervisors but for Ecto it is just the init function:
defmodule MyApp.Repo do
use Ecto.Repo, otp_app: :my_app
@doc """
Dynamically loads the repository url from the
DATABASE_URL environment variable.
"""
def init(_, opts) do
{:ok, Keyword.put(opts, :url, System.get_env("DATABASE_URL"))}
end
end
Which will hopefully push the community towards the proper path, as init is typically the correct place for such work.
8 Likes
Looks like step in the right direction. That’s the pain point, thanks for taking care!
Could you explain how on_init works?
What about custom variables/modules? e.g. i’ve installed some 3rd-party module and want to configure it. Or adding custom variables to current project.
It should be uniform as much as possible
1 Like
It invokes the given module+function+args passing the configuration as first argument. It must return the updated configuration.
Ideally every module or 3rd party application will either have a configuration API (which may be as simple as a proxy to Application.get_env) or an init callback you can customize. It is not uniform because the latter is about long running processes. If you change the port Phoenix is serving after Phoenix is started, it has no effect, therefore it belongs in init.
2 Likes
I am out my of my depth here but I don’t really see the problem with using application environment variables. They are very dynamically settable and at least when starting erlang you can give explicit values in your start command. Or include a specific config file which sets the environment variables. This means you can give use different config files for different cases, for example one for local testing and one for production.
Again I don’t know how to do this when starting elixir, either directly or through mix.
1 Like
Oh, the application environment works great and it is not being replaced in any way.
The problem is with system environment variables and the fact releases do not support them. So you cannot have config files that read from the system environment unless you use tools that overlays environments variables or pass them explicitly via the command line by editing the vm.args, which are both awkward. And even if you edit vm.args, you still can’t pass the system variables on appups or relups.
The system environment is only one of the many places where dynamic configuration may be read from. Some folks have json files written by the deployment system that they need to read on startup. So the goal is to provide general guidance on how to handle dynamic configuration. At the end of the day you can even read it from System.get_env and store it in the application environment. Our goal is to provide guidance in those situations.
1 Like
This is mostly the case where people want to read shell environment variables. There is an obscure reason for doing so: cloud platform providers such as Heroku do force users to do so.
And example is a PORT environment variable that has to be read from shell, in order to configure cowboy to bind to proper port. If you don’t do that, no requests will arrive to your application at all. The same issue is with database connection URL and many addons require that you read your config from shell ENV and configure at runtime rather than compile time. It’s not ideal but the way it is.
2 Likes
I understand that part of the problem, what I don’t understand is why you need to add a special hack to solve a problem which is easy to solve anyway with the existing system. Starting the system is very versatile and it is easy to specify multiple config files or args files, of which vm.args is an example, at start up time. These files are easy to generate and so as to read shell variables.
That is the bit I don’t understand.
2 Likes
Elixir works perfectly fine with erlang config files. The problem is that they differ significantly from what people are used to when it comes to deployments. People expect it to “just work™” in an ad-hoc way - that’s how most systems they’ve used worked, and generating a file on the fly isn’t what I’d call “just working”.
1 Like
This doesn’t work for development/testing/ci, unless you are building releases for those environments. Furthermore, it doesn’t compose well: if I have 3 applications in an umbrella, I would like to place each config close to the application, instead of having to remember to specify the configuration later on when building the release. And even if you edit vm.args, you still can’t pass the system variables on appups or relups.
Plus, some systems have their configuration in .json or other syntax that are written by their deployment system, how would you suggest them to read and parse those config files in vm.args?
Which part of reading configuration for a process in its init callback is a hack? Because the currently recommended approach is simply that. Or are you referring to the old {:system, ...} tuple?
1 Like
