Very slow compilation/xref

Yeah, I realize it’s somewhat ridiculous.

It’s a legacy application in which we are replacing some of the slower and more memory intensive ruby bits, and unfortunately the application has a couple thousand different dbs, due to a roll-your-own sharding scheme. Since Ecto can’t (as far as I can tell) have one Repo that connects to multiple dbs, this was the solution I came up with… and it’s actually working great, except for the slow compilation (which I can live with… I use a separate branch with just a few repos when doing heavy development).

When you say “without using any other libraries”, does that include Phoenix? I can replicate the issue with a brand new Phoenix app with only the default stuff, and just that one big repo file. The xref output includes just one extra line for that file.

Whoa, 2000 Repo files is… very big, each repo links to a variety of other Ecto things. use Ecto.Repo, ... is not at all lightweight. A question, why would you ever need that many repo’s, I can see needing a couple max, but that many? o.O?

Can you replicate the issue without using any other libraries? What is the xref graph results of linkages of the main library (not that 2000-repo test)?

You can just create a file with 2000 defmodules for repos into the lib folder of an existing Phoenix app (or create a new one) and see the difference.

For a new app, on my machine, it takes 1 second to compile a bare Phoenix app, and 135 seconds to compile it with that file in it.

Also, the “mix xref graph” time jumps from <1sec to about 55 seconds.

Here’s the ruby code I used to to create the file:

file_name = "repo_modules.ex"

code = <<-CODE
defmodule RepoTest.Repo__NDX__ do
  use Ecto.Repo, otp_app: :ebx, adapter: Ecto.Adapters.Postgres
  def init(_, opts) do
    {:ok, Keyword.put(opts, :url, System.get_env("DATABASE_URL"))}
  end
end
CODE

(1..2000).each{|ndx| File.open(file_name,"a"){|f| f.write(code.gsub(/__NDX__/,ndx.to_s))}}

Do you have an example project?

I suspect this is simply a question of there being a lot of modules, but I think there might be some things we could speed up.