Very slow compilation/xref

pedrogarrett

I have a Phoenix app that has its Postgres data split over about 2,000 different dbs.

Basically, a main db contains the information needed to connect to all of the other dbs… I’ve managed to get this working in Phoenix by statically creating 2,000 different modules that use Ecto.Repo, and dynamically starting up supervisors for all of them after the main Repo is started.

This works fine, but I’ve noticed that compilation now takes a couple of minutes (as opposed to less than 1 second with about 50 modules). That seems over-long… and mix xref always takes a couple of minutes as well. I’ve tried putting the 2,000 module defs into 1 file, and also into 2,000 different files, but it doesn’t seem to make any difference as far as compilation and xref time goes.

Is this just because a couple thousands modules is a lot and it will always take a while, or is there maybe something else going on?

Thanks in advance for any help or information.

4 comments

/phoenix #ecto #troubleshooting

1 1095 4

2018-05-04 21:21:09 UTC

Showing Posts 1 to 4

Show Best Posts
Show All (oldest first)
Show All (newest first)

michalmuskala

Do you have an example project?

I suspect this is simply a question of there being a lot of modules, but I think there might be some things we could speed up.

Post #1

pedrogarrett

You can just create a file with 2000 defmodules for repos into the lib folder of an existing Phoenix app (or create a new one) and see the difference.

For a new app, on my machine, it takes 1 second to compile a bare Phoenix app, and 135 seconds to compile it with that file in it.

Also, the “mix xref graph” time jumps from <1sec to about 55 seconds.

Here’s the ruby code I used to to create the file:

file_name = "repo_modules.ex"

code = <<-CODE
defmodule RepoTest.Repo__NDX__ do
  use Ecto.Repo, otp_app: :ebx, adapter: Ecto.Adapters.Postgres
  def init(_, opts) do
    {:ok, Keyword.put(opts, :url, System.get_env("DATABASE_URL"))}
  end
end
CODE

(1..2000).each{|ndx| File.open(file_name,"a"){|f| f.write(code.gsub(/__NDX__/,ndx.to_s))}}

Post #2

OvermindDL1

Whoa, 2000 Repo files is… very big, each repo links to a variety of other Ecto things. use Ecto.Repo, ... is not at all lightweight. A question, why would you ever need that many repo’s, I can see needing a couple max, but that many? o.O?

Can you replicate the issue without using any other libraries? What is the xref graph results of linkages of the main library (not that 2000-repo test)?

Post #3

pedrogarrett

Yeah, I realize it’s somewhat ridiculous.

It’s a legacy application in which we are replacing some of the slower and more memory intensive ruby bits, and unfortunately the application has a couple thousand different dbs, due to a roll-your-own sharding scheme. Since Ecto can’t (as far as I can tell) have one Repo that connects to multiple dbs, this was the solution I came up with… and it’s actually working great, except for the slow compilation (which I can live with… I use a separate branch with just a few repos when doing heavy development).

When you say “without using any other libraries”, does that include Phoenix? I can replicate the issue with a brand new Phoenix app with only the default stuff, and just that one big repo file. The xref output includes just one extra line for that file.

Post #4

— All posts loaded —