stefanchrobot

Database connection issues during deployment on Digital Ocean App Platform

Hey, recently I started to have issues during deployments to Digital Ocean App Platform. I’m deploying an Elixir release packed as a Docker image. During startup I’m often getting a database connection error:

[myapp] [2023-03-26 09:31:39] 09:31:39.044 [error] Could not create schema migrations table. This error usually happens due to the following:
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39]   * The database does not exist
[myapp] [2023-03-26 09:31:39]   * The "schema_migrations" table, which Ecto uses for managing
[myapp] [2023-03-26 09:31:39]     migrations, was defined by another library
[myapp] [2023-03-26 09:31:39]   * There is a deadlock while migrating (such as using concurrent
[myapp] [2023-03-26 09:31:39]     indexes with a migration_lock)
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39] To fix the first issue, run "mix ecto.create".
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39] To address the second, you can run "mix ecto.drop" followed by
[myapp] [2023-03-26 09:31:39] "mix ecto.create". Alternatively you may configure Ecto to use
[myapp] [2023-03-26 09:31:39] another table and/or repository for managing migrations:
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39]     config :myapp, MyApp.Repo,
[myapp] [2023-03-26 09:31:39]       migration_source: "some_other_table_for_schema_migrations",
[myapp] [2023-03-26 09:31:39]       migration_repo: AnotherRepoForSchemaMigrations
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39] The full error report is shown below.
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39] ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2483ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39]   1. Ensuring your database is available and that you can connect to it
[myapp] [2023-03-26 09:31:39]   2. Tracking down slow queries and making sure they are running fast enough
[myapp] [2023-03-26 09:31:39]   3. Increasing the pool_size (although this increases resource consumption)
[myapp] [2023-03-26 09:31:39]   4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39] See DBConnection.start_link/2 for more information
[myapp] [2023-03-26 09:31:39] 
[myapp] [2023-03-26 09:31:39]     (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:913: Ecto.Adapters.SQL.raise_sql_call_error/1
[myapp] [2023-03-26 09:31:39]     (elixir 1.14.3) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
[myapp] [2023-03-26 09:31:39]     (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:1005: Ecto.Adapters.SQL.execute_ddl/4
[myapp] [2023-03-26 09:31:39]     (ecto_sql 3.9.2) lib/ecto/migrator.ex:677: Ecto.Migrator.verbose_schema_migration/3
[myapp] [2023-03-26 09:31:39]     (ecto_sql 3.9.2) lib/ecto/migrator.ex:491: Ecto.Migrator.lock_for_migrations/4
[myapp] [2023-03-26 09:31:39]     (ecto_sql 3.9.2) lib/ecto/migrator.ex:403: Ecto.Migrator.run/4
[myapp] [2023-03-26 09:31:39]     (ecto_sql 3.9.2) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3
[myapp] [2023-03-26 09:31:39]     nofile:1: (file)

My migration task seems unable to connect to the database.

The DB exists and is reachable via other means,
The DB connection pool size is 22,
The app’s pool size is set to 6,
I’m only running one instance,
I use zero-downtime deployments, so max connections should be 12 during deployment.

I’m only experiencing this during deployments which makes them fail. Any idea what might be causing this?

20 comments

#deployment #database #digitalocean

13 3139 20

2023-10-19 13:00:37 UTC

Most Liked

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

@stefanchrobot really nice job debugging. One thing that may help here is instead of running your migrations as an eval task, put the migrator in your supervision tree via Ecto.Migrator — Ecto SQL v3.14.0. This way it isn’t competing with other activity in your application, and it will also block application start until the migrations can run.

Post #17

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Yeah we’ve been doing this for a long time since it synergizes really well with the K8s liveliness probes and readiness probes. Basically what we have is two endpoints /alive and /ready. /alive unconditionally returns true but /ready looks like this:

  def ready(conn, _params) do
    if Application.get_env(:myapp, :ready) do
      json(conn, %{ready: true})
    else
      send_resp(conn, 503, "")
    end
  end

Then our supervision tree looks like:

Phoenix Endpoint, #( this has `/alive` 200, but `/ready` is 503 still.)
DB Migrator,
Repo,
...Other Children,
MyApp.DeploymentNotifier, # this is just a one shot genserver that sets the env variable used in `/ready`

K8s has two different timeouts for pods. The first is about whether it’s alive at all, and the second is whether it is ready. It also only hooks up nodes to the load balancer that are /ready. This is K8s specific but this paradigm is found in other deployment structures as well. Basically it just ensures your app has time to initialize anything it needs before getting traffic, while still getting some indication from the app that it is alive and starting to boot.

Post #19

jerdew

Not sure if this is the same problem, but on DO I found I had to set the :maintenance_database to defaultdb (Ecto default is postgres) if I wanted to do create/migrate as part of a deploy.

Post #2