How to load test or prod config values dinamically into code?

I’m moving my project from testing to production, how do I switch between test and production environment variables—stuff like API URLs, sandbox modes, etc.

I’ve just been hardcoding test values, but now that I finished I need to be able to switch to production settings in prod, so I can keep a version to prod and one to test. I’m pretty sure there is a way to do that because that are a lot of config files, but I have no clue on how to do it.
to be more clear, I have a list of api’s to call declared on my config, and all of those have test versions, so I want to load the test version list when running test, and in prod load the prod list

Can you show how you’re currently doing it?

sure
this is on my config file

config :testespay, :blockchain_apis,
instantnodes: %{
  url: "https://api1.example.com",
  default_tokens: 10,
  refresh_time: 50000,
  circuit_breaker_threshold: 5
},
quicknode: %{
  url: "https://api2.example.com",
  default_tokens: 10,
  refresh_time: 50000,
  circuit_breaker_threshold: 5
},
ankr: %{
  url: "https://api2.example.com",
  default_tokens: 20,
  refresh_time: 50000,
  circuit_breaker_threshold: 5
}

and i’m loading into my code

defmodule Testespay.Genservers.ApiLoadBalancerBlockchain do
  use GenServer
  @my_apis Application.compile_env(:testespay, :blockchain_apis)  # Configurado em config.exs

  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, %{}, opts)
  end

  def init(_) do
    :ets.new(:api_states, [:set, :named_table, :public])

    registry_apis();
    print_all_apis()
    pick_api()
    {:ok, %{}}
  end

  # defp schedule_token_refresh, do: Process.send_after(self(), {:refill_tokens, 1000})

  defp registry_apis() do
    for {api_name, config} <- @my_apis do
      :ets.insert(:api_states, {
        api_name,
        %{
          tokens: config[:default_tokens],
          refresh_time: config[:refresh_time],
          oks: 0,
          errors: 0,
          last_refresh: System.os_time(:second),
          circuit_status: :closed
        }
        })
    end
  end




  def handle_call(:pick_api, _from, current_state) do
    api = pick_api()
    {:reply, api, current_state}
  end

  defp pick_api() do
    IO.puts("picking")
    api =
      :api_states
      |> :ets.tab2list()
      |> Enum.filter(fn {_name, data} -> available?(data) end)
      |> select_api_with_most_tokens()
      |> case do
        nil -> {:error, :no_apis_available}
        {name, _data} -> {:ok, name}
      end

    IO.puts("final api choose list")
    IO.inspect(api)
    api
  end

  defp available?(%{tokens: t, circuit_status: status, errors: e}) do
    t > 0 and status == :closed and e < 10
  end

 def select_api_with_most_tokens(list) do
  Enum.reduce(list, nil, fn {name, data}, acc ->
    case acc do
      nil -> {name, data}
      {_, %{tokens: max_tokens}} when data.tokens > max_tokens -> {name, data} ##retorna nome e data
      _ -> acc
    end
  end)
 end

  def handle_info({:perform_task, api_name, state}, current_state) do
    [{^api_name, current_state_map}] = :ets.lookup(:api_states, api_name)
    IO.inspect(current_state_map, label: "Current map for #{api_name}")

    updated_state = Map.put(current_state_map, :tokens, state.tokens)

    :ets.insert(:api_states, {api_name, updated_state})
    IO.puts("new tokens for #{api_name} reset to #{state.tokens}")

    Process.send_after(self(), {:perform_task, api_name, %{tokens: state.tokens, refresh_time: state.refresh_time}}, state.refresh_time)

    {:noreply, current_state}
  end

  defp print_all_apis() do
    :ets.tab2list(:api_states)
    |> Enum.each(fn {api_name, state} ->
      reduced_state = %{
        tokens: state.tokens,
        refresh_time: state.refresh_time
      }
      Process.send_after(self(), {:perform_task, api_name, reduced_state}, 0)
      IO.inspect({api_name, state}, label: "API State")
    end)
  end

end

Looks to be how you should be doing it in dev / test config files. In runtime.exs you could use environment variables or loading a YAML / TOML file.

and for now, the only commands I use to do my things are mix.test and mix phx.server

can you provide a schema on how it would be?

looks like the correct approach would be to declare the same variable name on prod.exs, but with the prod values I want, it’s that correct?

Nowadays not exactly. You can f.ex. use System.get_env! to make sure the app will not start if the env var is missing.

I can show you a small snippet when I’m back at the computer but it’s really not hard.

Or dotenv files. I prefer these since they can also be loaded in as Docker Compose environment variables.

sure it would be extrem helpfull

you use the env variable to load from your dotenv? something like
[prod-apis] …
[test-apis]…
so it comes down to just fetch the currently execution enviroment variable and use as key?

Just one random example:

# dev.exs
config :ueberauth, Ueberauth.Strategy.Google.OAuth,
  client_id: "stuff",
  client_secret: "more_stuff"

# runtime.exs
config :ueberauth, Ueberauth.Strategy.Google.OAuth,
  client_id: System.fetch_env!("GOOGLE_OAUTH_CLIENT_ID"),
  client_secret: System.fetch_env!("GOOGLE_OAUTH_SECRET")

Here’s a way to load a dotenv file for your current config env (e.g. dev, test, prod) by default, while allowing an optional override (e.g. if you want to use a custom dotenv file for whatever reason, I use this for testing in CI).

config/runtime.exs

import Config
import Dotenvy

dotenv_directory_path = Path.join([System.get_env("RELEASE_ROOT", "."), "envs"])
dotenv_filename = System.get_env("DOTENV_FILENAME", "#{config_env()}.env")
dotenv_file_path = Path.join(dotenv_directory_path, dotenv_filename)

source!(dotenv_file_path)

# Now add the rest of your runtime config...
  • When you run iex -S mix, it will load from envs/dev.env.
  • When you run mix test or MIX_ENV=test iex -S mix, it will load from envs/test.env.
  • In prod, it will load from envs/prod.env.
  • If you want to specify a custom dotenv file, you can run e.g. MIX_ENV=test DOTENV_FILE=some-file.env iex -S mix.

so I declare independently on each config file (dev, test, prod) and import everything through runtime based on the giving env right?
and in my code I keep the line

  @my_apis Application.compile_env(:testespay, :blockchain_apis)

?

For all the runtime stuff, you would use Application.get_env/3 (or fetch_env/2, fetch_env!/2) since you’re loading these values in runtime.

For compile-time stuff, you would use module attributes (And Application.compile_env/3) as in your example. If these values are coming from the values declared in runtime.exs, you can just bring them inside your functions and assign variables using fetch_env/get_env, e.g.:

def something do
  my_apis =
    Application.fetch_env!(:testespay, :blockchain_apis)
    |> Keyword.fetch!(:some_api)

  do_stuff()
end

Or use a getter function:

def get_blockchain_apis(api),
  do: Application.fetch_env!(:testespay, :blockchain_apis) |> Keyword.fetch!(api)

def something, do: get_blockchain_apis(:something) |> do_stuff()
1 Like

I could get those variables to work using runtime.exs pushing for other files, but couldnt do it with my database config, can yoy assist me on that as well?
Till now I was only using my config.exs


config :testespay, Testespay.Repo,
  log: :debug,
  database: "teste10",
  user: "pedri",
  password: "12345",
  hostname: "localhost",
  port: 5432

how can I load different specs through prod dev?
when running with prod the compiler asks for a url_source

** (RuntimeError) environment variable DATABASE_URL is missing.
For example: ecto://USER:PASS@HOST/DATABASE

instead of the usual config
I tried to put the url in the .env and expose it to mix

pedri@DESKTOP-TQNI8GF:~/devmain$ source .env
pedri@DESKTOP-TQNI8GF:~/devmain$ MIX_ENV=prod PHX_SERVER=true mix phx.server

and nothing changes


** (RuntimeError) environment variable DATABASE_URL is missing.
For example: ecto://USER:PASS@HOST/DATABASE

    /home/pedri/devmain/config/runtime.exs:26: (file)
    (stdlib 5.2.1) erl_eval.erl:750: :erl_eval.do_apply/7
    (stdlib 5.2.1) erl_eval.erl:494: :erl_eval.expr/6
    (stdlib 5.2.1) erl_eval.erl:136: :erl_eval.exprs/6
    (elixir 1.17.0) lib/code.ex:572: Code.validated_eval_string/3

You still need the existing syntax that was in your config/runtime.exs file before. But with Dotenvy, you can then extend it to work with your dotenv files. You can fall back to default values, or fail if no value found, or any combination that works for your needs. The world is your oyster.

e.g. Here is what my repo config looks like after integrating Dotenvy:

# Above this line would be the Dotenvy 'source' logic from my first post in this thread

# Check for environment variable, or require value from dotenv
database_name = System.get_env("POSTGRES_DB") || env!("POSTGRES_DB", :string!)

config :my_project, MyProject.Repo,
  # Raise exception if user and password not set in dotenv file
  username: env!("POSTGRES_USER", :string!),
  password: env!("POSTGRES_PASSWORD", :string!),
  database:
    if(config_env() == :test,
      # Check system environment only for MIX_TEST_PARTITION (default Phoenix behaviour)
      do: database_name <> System.get_env("MIX_TEST_PARTITION", ""),
      else: database_name
    ),
  # Check for environment variable, or require value from dotenv file
  hostname: System.get_env("POSTGRES_HOST") || env!("POSTGRES_HOST", :string!),
  # Check for value in dotenv file, or fall back to default value
  port: env!("POSTGRES_PORT", :integer, 5432)

I recommend taking a look through the Dotenvy docs, and maybe starting with a throwaway project so you can see how to integrate the syntax. Remember: Dotenvy extends your runtime config, it doesn’t replace it.

One thing I would like to add is that there is no need to mix the system environment variables with the values defined in your dotenv file. It can be done, but it’s not required. The config can just be read from the file into your runtime config without interacting with the system environment (via Dotenvy.source!).