Manually Load an Elixir Config File at Runtime

We run in ECS and use param store for our secrets. You can simply set environment variables in your task definition under secrets.

"containerDefinitions": [
      "secrets": [
        {
          "valueFrom": "/param/path",
          "name": "VARIABLE_NAME"
        }
]

from there I put them in prod.exs as configuration.

I know valueFrom supports secret manager too.

EDIT: Pass sensitive data to an Amazon ECS container - Amazon Elastic Container Service

I believe your solution is more common and simpler than the example I gave. However, this is similar to my third point of what I dislike about alternatives; it’s a roundabout way to load secrets into your application. You have to have an intermediary step between fetching that configuration and loading it; in your case, it’s having to configure those variable names in ECS and then extract them in your application configuration.

I don’t think you’ll find a magic bullet here. You have to do the work somewhere. And in the case of Config, it’s a compile-time concern. If you want runtime config, you have to write the code or do something similar to what I explained.

Rarely in programming would I argue that anything is a magic bullet - there are tradeoffs in every decision we make! I’m just looking for a discussion about this idea I had and why/how I should/shouldn’t do it.

Sorry, I should have made myself clearer: I’m running this application in production as an Elixir release. Therefore I can write a custom Config.Provider to load application configuration at runtime. Do you happen to build and run your application(s) as an Elixir release?

Oh I’m willing to brainstorm. I’m completely isolated and bored just like you!

I do run as a release.

I actually just created a toy app to investigate possibilities right now.

So I’ve thought about it some more and looked at some docs in the Code module and there’s a warning about evaluating code coming from the network. Dang, seems like loading data from SecretsManager would be a bad idea from a security perspective…

Still - could there be another way??

I want to see if it’s possible. My current solution would be to have the config provider do:

defmodule MyConfigProvider do
  alias ExAws.SecretsManager, as: Secrets

  def init(secret_id) when is_binary(secret_id), do: secret_id

  def load(config, secret_id) do
    %{"SecretValue" => raw_elixir_code} = Secrets.get_secret_value(secret_id) |> ExAws.request!()
    {config_to_merge, _} = Code.eval_string(raw_elixir_code)
    Config.Reader.merge(config, config_to_merge)
  end
end

I have the same idea

defmodule SecretsConfigProvider do
  @behaviour Config.Provider

  def init(path) when is_binary(path), do: path

  def load(config, path) do
    {:ok, _} = Application.ensure_all_started(:jason)

    custom_config =
      path
      |> File.read!()
      |> Jason.decode!(keys: :atoms)
      |> resolve_config()

    Config.Reader.merge(config, custom_config)
  end

  defp resolve_config(config, acc \\ []) do
    Enum.into(config, [], fn
      {key, value} when is_map(value) -> {key, resolve_config(value)}
      {key, value} -> {key, read_secret(value)}
    end)
  end

  @aws [region: "us-west-1"]

  defp read_secret(secret) do
    %{"Parameter" => %{"Value" => value}} =
      secret
      |> ExAws.SSM.get_parameter()
      |> ExAws.request!(@aws)

    value
  end
end

and then load with a json file:

{
  "app": {
    "a": "/insurance/settlement/jobs_url",
    "b": {
      "c": "/notifications/api/url"
    }
  }
}

like

iex(1)> SecretsConfigProvider.load([existing: :config, app: [d: :d]], "/Users/martin/code/tmp/runtime_config/priv/secrets.json")
[
  existing: :config,
  app: [
    d: :d,
    a: "https://REDACTED.execute-api.us-west-1.amazonaws.com/staging",
    b: [c: "https://REDACTED.com"]
  ]
]

I believe this would work and is totally feasible.

…and I may just start using this instead of task definition secrets!

EDIT: Fixed the duplicate issue

Ohhh!

I think I understand your question in more detail now. You want to actually store the config file in secrets manager and eval it! I don’t think I would try that. Seems odd to me.

Yeah… The security concern is what makes me want to avoid it. Using dynamic code loading in a sandboxed environment for, say, integration tests, can be really useful. But… doing the same thing in a production environment - with data from a remote source - probably isn’t a good idea.

Yes yes yes, you’re exactly right! The rel/releases.exs file is super useful for loading some information at boot-time, but it’s packaged in with the release. Essentially what I’m looking for is to fetch that file from a remote source at boot-time!

I’ve been thinking about this.

Since you trust yourself, and therefore your own code, I think if you ditch the Mix.Config stuff, and just define a keyword list, you will be able to safely eval the code using your config provider.