Preparing for Production

I don’t use Swarm, but your problem seems like a more general case where secrets must be provided when starting the app (as opposed to building the release), and they might need to be derived from various sources, not necessarily OS env. In my cases, I used etcd and custom json files to fetch stuff, so various OS env impros wouldn’t work for me anyway.

The way I deal with this (which doesn’t work for everything, but does for most things, including Phoenix and Ecto), is:

1. In application start callback, prior to starting the top-level supervisor, I fetch secrets from wherever.
2. I merge secrets into proper places in app env (e.g. repo or endpoint config).
3. I start the supervision tree.

Thanks, @sasajuric. This seems like a good idea.

Would you just leave this blank, or not even defined?

config :app, database_password: "" 

Then in start, I could do File.ls |> Enum.each |> merge_config with a function for each parameter I expect?

Useful suggestion, thanks!

Would you just leave this blank, or not even defined?

I leave these settings undefined. In case you use those settings only in prod, I suggest defining them in dev and test only.

Well, in start you fetch those parameters from wherever they are, and then merge them, yes. AFAIK there’s no “merge env” function, so I do it by reading the top-level setting (say MyApp.Repo), adding additional info (e.g. password) into it, and putting it back with Application.put_env.

Awesome. Thank you very much for your help, it’s appreciated muchly

Hi @sasajuric, do you have any examples of doing this? I’ve been trying for a couple of hours now.

get_all_env(app) returns a List of tuples, which is in-turn List of List and I have no idea how to “merge”

get_env(app, key) also returns a List and stumps me too.

A simple approach:

def merge(a, b), do: Keyword.merge(a, b, &resolve_conflict/3)
defp resolve_conflict(_key, v1, v2) do
    cond do
      Keyword.keyword?(v1) && Keyword.keyword?(v2) ->
        Keyword.merge(v1, v2, &resolve_conflict/3)
      :else ->
        v2
    end
end

This will recursively merge two keyword lists, taking the newer value when conflicts occur, except when the value is a keyword list, in which case it merges. This obviously is a bit of an over simplification, but improving it is an exercise left to the reader

Let’s say I want to set a repo password:

repo_settings = Application.get_env(:my_app, MyApp.Repo)
modified_repo_settings = Keyword.merge(repo_settings, password: super_secret_password)
Application.put_env(:my_app, MyApp.Repo, modified_repo_settings)

If you need to do this for a couple of different top-level settings, its worth making a helper fun which could be used as:

update_app_env(:my_app, MyApp.Repo, &Keyword.merge(&1, password: super_secret_password))

The implementation is left as an exercise

Thanks again for everyone’s help! I’m getting close.

My “last” problem, I hope, is that this loading happens after KafkaEx starts, which is now failing due to missing configuration.

So far I’ve got the following:

def start(_type, _args) do                                                                                                                                                            
  App.Config.Loader.load_configuration() 
  ... super visor stuff
end

defmodule App.Config.Loader do
  require Logger

  def load_configuration() do
    with {:ok, files} <- File.ls("/run/secrets") do
      Enum.each(files, fn (file_name) ->
        load_configuration(file_name)
      end)
    end

  defp load_configuration(key = "api_hostname", hostname) do
    {:app_api, App.Api.Endpoint, [url: [host: hostname]]}
  end

  defp load_configuration(key = "secret_key_base", secret_key_base) do
    {:app_api, App.Api.Endpoint, [secret_key_base: secret_key_base]}
  end

  defp load_configuration(key = "json_web_token_secret", json_web_token_secret) do
    {:guardian, Guardian, [secret_key: json_web_token_secret]}
  end

  defp load_configuration(key = "elasticsearch_uri", uri) do
    {:tirexs, :uri, uri}
  end

  defp load_configuration(key = "elasticsearch_index", index) do
    {:app_api, :elasticsearch_index, index}
  end

  defp load_configuration(key = "kafka_hosts", host_string) do
    {:kafka_ex, :brokers, host_string
    |> String.split(",")
    |> Enum.map(fn(uri) ->
      [host, port] = String.split(":")
      {host, String.to_integer(port)}
    end)}
  end

  defp load_configuration(key = "youtube_api_key", youtube_api_key) do
    if ! is_nil(youtube_api_key) do
      {:tubex, Tubex, [api_key: youtube_api_key]}
    end
  end

  defp load_configuration(key, _) do
    Logger.warn("External configuration (#{key}) provided, but not loaded")
  end
end

Is it normal for applications to be started before my “primary”?

Applications should be loaded in dependent order, so if you depend on it then it will be up before yours. Libraries that are designed to be loaded after yours (should) give you a supervisor to place in to your own supervision tree so you can then do whatever you want to their loading, just like how Phoenix’s Endpoint works.