Manually Load an Elixir Config File at Runtime

Questions / Help
, ,
1

I’d like to discuss how to manually load Elixir configuration files at runtime.

Macro Intent

I want to load an elixir config file (something that looks like, for example, config/config.exs) at runtime, and load that into the application configuration.

Micro Intent

Ultimately, I want to write a Config.Provider implementation which fetches the contents of a secret stored in AWS secrets manager. I’d like for those contents to be an Elixir script with import Config at the top, followed by configuration defined the same way you would in a project’s config/config.exs file. I then want to load that configuration into the current (running!) application’s configuration.

Questions & Discussion

  • Let’s assume that I can load the secret configuration file from anywhere into a binary with an arbitrary Config.Provider implementation (ignore SecretsManager for now). How would I go about compiling it and loading it into the application configuration?
  • What do you think of this approach to secret management, at least in theory?
  • What (at least roughly) do you do for secret management in your Elixir and/or Phoenix applications?

My Take

Let’s consider an alternative Config.Provider implementation: instead load a YAML file, parse it, and translate it into a keyword list to merge into the current config. In my eyes that has a few downsides that I dislike:

  1. Added dependency on a YAML parser
  2. Designing how that YAML is structured is not a trivial task
  3. It’s a roundabout way to get to what I ultimately want, which is just to load configuration from a remote location into the current node’s application configuration.

These are my thoughts. Let me know yours!

1 Like
2

We run in ECS and use param store for our secrets. You can simply set environment variables in your task definition under secrets.

"containerDefinitions": [
      "secrets": [
        {
          "valueFrom": "/param/path",
          "name": "VARIABLE_NAME"
        }
]

from there I put them in prod.exs as configuration.

I know valueFrom supports secret manager too.

EDIT: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html

1 Like
3

I believe your solution is more common and simpler than the example I gave. However, this is similar to my third point of what I dislike about alternatives; it’s a roundabout way to load secrets into your application. You have to have an intermediary step between fetching that configuration and loading it; in your case, it’s having to configure those variable names in ECS and then extract them in your application configuration.

4

I don’t think you’ll find a magic bullet here. You have to do the work somewhere. And in the case of Config, it’s a compile-time concern. If you want runtime config, you have to write the code or do something similar to what I explained.

5

Rarely in programming would I argue that anything is a magic bullet - there are tradeoffs in every decision we make! I’m just looking for a discussion about this idea I had and why/how I should/shouldn’t do it.

Sorry, I should have made myself clearer: I’m running this application in production as an Elixir release. Therefore I can write a custom Config.Provider to load application configuration at runtime. Do you happen to build and run your application(s) as an Elixir release?

6

Oh I’m willing to brainstorm. I’m completely isolated and bored just like you! :stuck_out_tongue:

I do run as a release.

I actually just created a toy app to investigate possibilities right now.

1 Like
7

So I’ve thought about it some more and looked at some docs in the Code module and there’s a warning about evaluating code coming from the network. Dang, seems like loading data from SecretsManager would be a bad idea from a security perspective…

Still - could there be another way??

I want to see if it’s possible. My current solution would be to have the config provider do:

defmodule MyConfigProvider do
  alias ExAws.SecretsManager, as: Secrets

  def init(secret_id) when is_binary(secret_id), do: secret_id

  def load(config, secret_id) do
    %{"SecretValue" => raw_elixir_code} = Secrets.get_secret_value(secret_id) |> ExAws.request!()
    {config_to_merge, _} = Code.eval_string(raw_elixir_code)
    Config.Reader.merge(config, config_to_merge)
  end
end
1 Like
8

I have the same idea

defmodule SecretsConfigProvider do
  @behaviour Config.Provider

  def init(path) when is_binary(path), do: path

  def load(config, path) do
    {:ok, _} = Application.ensure_all_started(:jason)

    custom_config =
      path
      |> File.read!()
      |> Jason.decode!(keys: :atoms)
      |> resolve_config()

    Config.Reader.merge(config, custom_config)
  end

  defp resolve_config(config, acc \\ []) do
    Enum.into(config, [], fn
      {key, value} when is_map(value) -> {key, resolve_config(value)}
      {key, value} -> {key, read_secret(value)}
    end)
  end

  @aws [region: "us-west-1"]

  defp read_secret(secret) do
    %{"Parameter" => %{"Value" => value}} =
      secret
      |> ExAws.SSM.get_parameter()
      |> ExAws.request!(@aws)

    value
  end
end

and then load with a json file:

{
  "app": {
    "a": "/insurance/settlement/jobs_url",
    "b": {
      "c": "/notifications/api/url"
    }
  }
}

like

iex(1)> SecretsConfigProvider.load([existing: :config, app: [d: :d]], "/Users/martin/code/tmp/runtime_config/priv/secrets.json")
[
  existing: :config,
  app: [
    d: :d,
    a: "https://REDACTED.execute-api.us-west-1.amazonaws.com/staging",
    b: [c: "https://REDACTED.com"]
  ]
]

I believe this would work and is totally feasible.

…and I may just start using this instead of task definition secrets! :wink:

EDIT: Fixed the duplicate issue

1 Like
9

Ohhh!

I think I understand your question in more detail now. You want to actually store the config file in secrets manager and eval it! I don’t think I would try that. Seems odd to me.

10

Yeah… The security concern is what makes me want to avoid it. Using dynamic code loading in a sandboxed environment for, say, integration tests, can be really useful. But… doing the same thing in a production environment - with data from a remote source - probably isn’t a good idea.

Yes yes yes, you’re exactly right! The rel/releases.exs file is super useful for loading some information at boot-time, but it’s packaged in with the release. Essentially what I’m looking for is to fetch that file from a remote source at boot-time!

11

I’ve been thinking about this.

Since you trust yourself, and therefore your own code, I think if you ditch the Mix.Config stuff, and just define a keyword list, you will be able to safely eval the code using your config provider.

12

Yeah in toy projects I’ll probably give it a wack, why not? Dynamic code loading is useful!

I’ll have to do much more research on it before I can do it in a work setting though…ha

13

If you check that the code you get from your secret manager is signed with a non-revoked key and you have reasonable degree of assurance in your intrusion detection, evaluating signed code is perfectly fine. World runs on evaluating signed code.

It’s all about threat model, right? If your threat model is “my secret storage got compromised”, odds are, arbitrary code execution on your production servers has already happened by then.

But yeah, awesome thread, we were getting secrets one by one on bootstrap, and are currently moving to evaluating one signed secret instead.