anthonator

anthonator

Sharing a DBConnection pool with multiple Ecto Repo's

Is it possible to create a shared connection pool that can be used across many repos?

Most Liked

unnawut

unnawut

@anthonator Just dropping by to say that your approach inspired me to figure out something similar.

I did just slightly differently without a new connection pool module (and thus no delegations).

defmodule DB.SharedConnectionPool do
  alias DBConnection.ConnectionPool

  def child_spec({mod, opts}) do
    opts = Keyword.put_new(opts, :name, pool_name(opts))
    Supervisor.Spec.worker(__MODULE__, [{mod, opts}])
  end

  def start_link({mod, opts}) do
    case GenServer.start_link(ConnectionPool, {mod, opts}, start_opts(opts)) do
      {:ok, pid} -> {:ok, pid}
      {:error, {:already_started, pid}} -> {:ok, pid}
      error -> error
    end
  end

  defp pool_name(opts) do
    case Keyword.fetch(opts, :shared_pool_id) do
      {:ok, pool} -> String.to_atom("#{__MODULE__}-#{pool}")
      :error -> __MODULE__
    end
  end

  # Exact same as `DBConnection.ConnectionPool`
  defp start_opts(opts) do
    Keyword.take(opts, [:name, :spawn_opt])
  end
end

And I have the repos configured like this:

config :my_app, MyApp.Repo,
  pool: DB.SharedConnectionPool,
  shared_pool_id: :my_shared_pool_id

config :my_second_app, MySecondApp.Repo,
  pool: DB.SharedConnectionPool,
  shared_pool_id: :my_shared_pool_id

This way makes it also possible to specifically configure what repos should share a pool, and multiple shared pools can be created.

The downside is that it still relies on the connection info from the configs, which means if two repos with different configs share the same pool, there’s a race condition and one repo config will be taken to start the pool. But otherwise this approach has the least impact on how Ecto works.


Full code here: Allow DB connection pool sharing between repos by unnawut · Pull Request #853 · omgnetwork/ewallet · GitHub (disclaimer: I’m employed by this project)

dylan-chong

dylan-chong

With db_connection 2.4.0, i get this error

** (UndefinedFunctionError) function Solve.Utils.DB.SharedConnectionPool.checkout/3 is undefined or private
    (utils 0.1.0) Solve.Utils.DB.SharedConnectionPool.checkout(#PID<0.13239.0>, [#PID<0.13592.0>, #PID<0.13591.0>], [log: #Function<14.6198013/1 in Ecto.Adapters.SQL.with_log/3>, source: "query_history_v1", timeout: 15000, pool_size: 10, pool: Solve.Utils.DB.SharedConnectionPool])
    (db_connection 2.4.0) lib/db_connection.ex:1082: DBConnection.checkout/3
    (db_connection 2.4.0) lib/db_connection.ex:1407: DBConnection.run/6
    (db_connection 2.4.0) lib/db_connection.ex:574: DBConnection.parsed_prepare_execute/5
    (db_connection 2.4.0) lib/db_connection.ex:566: DBConnection.prepare_execute/4
    (ecto_sql 3.5.4) lib/ecto/adapters/sql.ex:690: Ecto.Adapters.SQL.execute!/4
    (ecto_sql 3.5.4) lib/ecto/adapters/sql.ex:682: Ecto.Adapters.SQL.execute/5
    (ecto 3.5.6) lib/ecto/repo/queryable.ex:224: Ecto.Repo.Queryable.execute/4
    (ecto 3.5.6) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (saved_query_service 0.1.0) lib/query_service.ex:55: Solve.SavedQueryService.fetch_history/2
    (query_runner 0.1.0) lib/query_runner/tidy_history_task.ex:7: Solve.QueryRunner.TidyHistory.run/0
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Adding:

  defdelegate checkout(pool, callers, opts), to: ConnectionPool

to the connection pool here Sharing a DBConnection pool with multiple Ecto Repo's - #5 by unnawut seems to fix the issue

mbuhot

mbuhot

This is the reason I just share a repo between all umbrella apps, and declare the schemas separately in each app. Otherwise you can easily max out the 200 connection limit on an AWS db.t2.small instance :grinning_face_with_smiling_eyes:

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New

Other popular topics Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30877 112
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

We're in Beta

About us Mission Statement