Can't run umbrella project migration on Heroku

Hello! I am running an umbrella app on Heroku (http://www.notefile.io), but have not been able to run migrations there. Locally the app, db, and migrations are working. On heroku, all is well except that I cannot run migrations. This problem arose after I moved to the umbrella architecture. In order to run on Heroku, I did have to reduce the pool size (I chose POOL_SIZE=10 for heroku config and set it t the same in both config/dev.exs and config/prod.exs.) Before making these changes to POOL_SIZE, I got the too many connections error and the app would not run.

There are many errors below as a result of running heroku run POOL_SIZE=2 mix ecto.migrate . However, the first one to solve is the too many connections error. Then I will see if the others are spurious or not.

jxxmbp:ns_umbrella carlson$ heroku run POOL_SIZE=2 mix ecto.migrate
Running POOL_SIZE=2 mix ecto.migrate on ⬢ notefile... up, run.8424 (Hobby)

15:32:59.488 [error] Postgrex.Protocol (#PID<0.237.0>) failed to connect: ** (Postgrex.Error) FATAL 53300 (too_many_connections): too many connections for role "nmtqncpzsfkmbx"

15:32:59.496 [error] Could not retrieve migrated versions. This error typically happens when the "schema_migrations" table, which Ecto uses for storing migrationinformation, is already used by another library or for other purposes.

You can fix this by running `mix ecto.drop` in the appropriate `MIX_ENV` to drop the existing database and let Ecto start a new one with a proper definition of "schema_migrations" or by configuring the repository to use another source:

    config :lookup_phoenix, LookupPhoenix.Repo,
      migration_source: "some_other_table_for_schema_migrations"

The full error is shown below.

** (DBConnection.ConnectionError) connection not available because of disconnection
    (db_connection) lib/db_connection.ex:926: DBConnection.checkout/2
    (db_connection) lib/db_connection.ex:742: DBConnection.run/3
    (db_connection) lib/db_connection.ex:1133: DBConnection.run_meter/3
    (db_connection) lib/db_connection.ex:584: DBConnection.prepare_execute/4
    (ecto) lib/ecto/adapters/postgres/connection.ex:93: Ecto.Adapters.Postgres.Connection.execute/4
    (ecto) lib/ecto/adapters/sql.ex:243: Ecto.Adapters.SQL.sql_call/6
    (ecto) lib/ecto/adapters/sql.ex:193: Ecto.Adapters.SQL.query!/5
    (ecto) lib/ecto/adapters/postgres.ex:86: anonymous fn/4 in Ecto.Adapters.Postgres.execute_ddl/3

Note added (1)

I also tried this:

$  heroku maintenance:on
$  heroku run POOL_SIZE=2 mix ecto.migrate
$  heroku maintenance:off

However, I got the same set of errors as above, beginning with failed to connecttoo_many_connections.

Note added (2)

Another data point – the above failures are from a state of zero connections (apparently):

jxxmbp:ns_umbrella carlson$ heroku pg:info
=== HEROKU_POSTGRESQL_BRONZE_URL
Plan:        Hobby-dev
Status:      Available
Connections: 0/20
PG Version:  9.6.1
Created:     2017-04-21 11:14 UTC
Data Size:   8.4 MB
Tables:      3
Rows:        872/10000 (In compliance)
Fork/Follow: Unsupported
Rollback:    Unsupported
Add-on:      postgresql-colorful-71532

You need to set phoenix to use one less connection and run the migrate command with POOL_SIZE=1.

Thanks – however, I don’t believe this will work – the POOL_SIZE limit for my app on Heroku is 20. I did try heroku run POOL_SIZE=1 mix ecto.migrate with phoenix still set to use 10 of the 20 connections. Got the same error about too many active connections.

The problem arose when I switched from an ordinary app to an umbrella app. Heroku won’t help me on this because they do not support Elixir. Any other thoughts on this one?

Note added. I did try heroku run POOL_SIZE=1 mix ecto.migrate with a POOL_SIZE=9 for the app. Got the same error as before.

How many of the umbrella apps use Ecto?

I believe just one – the phoenix app. But I will check when i get home.

Long shot, but perhaps the web app is taking up all available connections? You can temp disable it with: heroku ps:scale web=0

2 Likes

That was it – thankyou!