Dynamically start repos

Hi everybody,
I have an application responsible to update a few repos the same way. I have MyApp.Repo and i will start it with different names and config. The number of the repos and their configuration are unknown at compile time. When the application starts it must read a config file with the list of the repos, and start them. Is there an elixir tool for this? I saw Config.Provider and Config.Reader. But it seems to be used for releases and i don’t know how to test it without releases and how to use it properly.
Thanks

Hey @Laetitia, the Ecto.Repo documentation describes this use case wonderfully in the Replicas and dynamic repositories guide. Does this guide answer your question already?

1 Like

Hey
Thank you for the answer. I was intended to use this great feature. I just don’t figure out how to configure the dynamic repos. It will be something like

Enum.map(repos, &MyApp.Repo.start_link(name: &1.name, url: &1.url)

where name and url have to be loaded on runtime. The number of repo is also known at runtime.
How to handle the config is the missing piece of my puzzle.

To create a repo at runtime you will need something like the following (not tested):

def define_repo(name, options, binding \\ [], env \\ []) do  
  ecto_repo = 
   quote do
     defmodule unquote(name) do
       use Ecto.Repo, unquote(options)
     end
   end
   
   Code.eval_quoted(ecto_repo, binding, env)
end
1 Like

Yes i can to it this way. But it does not solve my problem or i missed something.

Currently, in the runtime.exs config file, i have something like this:

config :my_app, MyApp.Repo,
url: System.fetch_env!(“DATABASE_URL”),
pool_size: String.to_integer(System.get_env(“POOL_SIZE”) || “10”)

i would like to have a config like that:

config :my_app, repos,
[ %{url: System.fetch_env!(“DATABASE_URL1”), pool_size: String.to_integer(System.get_env(“POOL_SIZE1”) || “10”)}.
%{url: System.fetch_env!(“DATABASE_URL2”), pool_size: String.to_integer(System.get_env(“POOL_SIZE2”) || “10”)}, etc…]

and i wold like to be able to change this config, only restart the app, and magic!, all the repos are build and started under the supervisor.

How could i do it? Any idea? Is there a bit of solution like the Config.Provider. If yes, i need more info how to achieve this.

Thank you all

You could do this with something like:

config :my_app, MyApp.Repos, [...list of repos...]

in runtime.exs and then code to handle that config in MyApp.Application.start:

  use Application

  def start(_type, _args) do
    children = [
      MyApp.Repo, # the default repo
      repo_specs(),
      # more things
    ] |> List.flatten()

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  defp repo_specs do
    Application.get_env(:my_app, MyApp.Repos)
    |> Enum.map(fn options ->
      # produce a child spec, based on options. Something like:
      {MyApp.Repo, options}
      |> Supervisor.child_spec(id: somehow_make_a_unique_id(options))
    end)
end

(the generators add most of this code to the file already)

1 Like

Thanks @al2o3cr!

i think I missed something. Do you mean i could change runtime.exs before starting the release, in order to add a repo for example? Techinally , i think so but is it conceptually right and a good practice?

Hmm, okay, if you don’t need to start a new Repo while the app is running then maybe this will help you:

# runtime.exs

databases = [
  First: [url: System.get_env("REPO_1_URL"), pool_size: System.get_env("REPO_1_POOL_SIZE")],
  Second: [url: System.get_env("REPO_2_URL"), pool_size: System.get_env("REPO_2_POOL_SIZE")],
]

config :my_app, :databases, databases

for {name, info} <- databases do
  config :my_app, Module.concat(Repo, name),
    url: info[:url],
    pool_size: info[:pool_size],
    ssl: true
end

This will define 2 Repos (FirstRepo and SecondRepo) for your application.

# application.ex
defmodule MyApp.Application do

  use Application

  def start(_type, _args) do
    children =
      [
        ... other children like Endpoint, PubSub, etc.
      ] ++ repos()


    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  def repos do
    for {name, _opts} <- Application.get_env(:my_app, :databases) do
      defmodule Module.concat(Repo, name) do
        use Ecto.Repo,
          otp_app: :my_app,
          adapter: Ecto.Adapters.Postgres,
          read_only: true
      end

      Module.concat(Repo, name)
    end
  end
end

This will then set the config for the 2 repos and start them.

If you change your config through the environment variables for your repos, you need to restart your application and it will apply the config changes to your repos.

Did this help you further?

4 Likes

Thank you! But how would you add a repo? Restart the app is not a problem. I just don’t want to recompile the code and build a new release in order to add a repo. If it is right to do it, it would be great to only have to edit the runtime.exs file. Is it the right way?

Once you build a release you cannot edit runtime.exs, that’s why it was suggested:

So, you get each repo configuration from the environment variables, therefore this means you cannot have a dynamic number of repos as you pretend.

For what I understand sometimes you may want to have 2 repos and other times have 200. You can use the code provided by @PJUllrich to achieve it, by having a environment variable with the total of repos to configure, like TOTAL_REPOS and then loop them in order to get the environment values for each System.get_env("REPO_#{index}_URL"), meaning that in the server environment you would need to set each of this vars REPO_1_URL to REPO_200_URL .

Something in the lines of:

total_repos = System.get_env("TOTAL_REPOS")

databases = Enum.map 1..total_repos, fn index -> [url: System.get_env("REPO_#{index}_URL"), pool_size: System.get_env("REPO_#{index}_POOL_SIZE")] end

Code not tested.

I hope it makes sense to you.

2 Likes

Hi @Exadra37, thank you and thank to all who tried to help.
Your answer makes a lot of sense. TOTAL_REPOS is a really good idea!

1 Like

Was a great idea, but built on top of the awesome work of @PJUllrich, that deserves 99% of the credit :slight_smile:

3 Likes

I’m a bit late to the party, but I just wanted to suggest one other possible improvement: instead of reading database config from environment variables, you could have your CI/build process create a file specifying the databases’ configuration, then read from that in your runtime.exs, something like:

# runtime.exs
config_path = "/tmp/databases.json"
config_json = File.read!(config_path)
config = Jason.parse!(config_json)

databases = Enum.map(config, fn database ->
  [url: database["url"], ...]
end)

If the dynamic database config is known when you run mix release, you can include it in priv/databases.json and read from it with Application.app_dir(:my_app, "priv"). Otherwise, just copy the config to /tmp/databases.json (or wherever) and read it from there.

This isn’t strictly any better/worse than environment variables, just another potential approach.

2 Likes