How to run initial db migration on Kubernetes/Docker setup

Questions / Help
1

I am having trouble trying to deploy my Phoenix app in our internal Kubernetes infra. Particularly, I’m not sure how to run the initial db migration. This is the logs on the pod.

04:41:47.447 [error] Could not create schema migrations table. This error usually happens due to the following:

  * The database does not exist
  * The "schema_migrations" table, which Ecto uses for managing
    migrations, was defined by another library
  * There is a deadlock while migrating (such as using concurrent
    indexes with a migration_lock)

To fix the first issue, run "mix ecto.create" for the desired MIX_ENV.

To address the second, you can run "mix ecto.drop" followed by
"mix ecto.create", both for the desired MIX_ENV. Alternatively you may
configure Ecto to use another table and/or repository for managing
migrations:

    config :acme, Acme.Repo,
      migration_source: "some_other_table_for_schema_migrations",
      migration_repo: AnotherRepoForSchemaMigrations

The full error report is shown below.

** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2993ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enough
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

    (ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1
    (elixir 1.16.0) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
    (ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1161: Ecto.Adapters.SQL.execute_ddl/4
    (ecto_sql 3.11.1) lib/ecto/migrator.ex:755: Ecto.Migrator.verbose_schema_migration/3
    (ecto_sql 3.11.1) lib/ecto/migrator.ex:563: Ecto.Migrator.lock_for_migrations/4
    (ecto_sql 3.11.1) lib/ecto/migrator.ex:432: Ecto.Migrator.run/4
    (ecto_sql 3.11.1) lib/ecto/migrator.ex:170: Ecto.Migrator.with_repo/3
    nofile:1: (file)
  • I am using MariaDB 10.22 with :myxql driver.
  • I have tried playing around the value of pool_size, queue_target, queue_internal of the db but no luck.
  • I can confirm that the db connection is valid and db exists.
  • My Dockerfile is the standard one generated with mix phx.gen.release --docker

The only thing I modified is this part:
CMD ["/app/bin/server"]

to this:
CMD ["sh", "-c", "/app/bin/migrate && /app/bin/server"]

Anyone with experience deploying a Phoenix app in Docker/Kubernetes can share their insight/config?

2

I have a separate script which sets up environmental variables, runs migrations, and then finally starts /app/bin/server. Not sure if this is the “correct” way to do it, but it was the simplest for me.

Partial startup script:

#!/bin/bash

# allocate swap space
if [ $UID -eq 0 ]; then
  fallocate -l 1024M /swapfile
  chmod 0600 /swapfile
  mkswap /swapfile
  echo 10 > /proc/sys/vm/swappiness
  swapon /swapfile
  echo 1 > /proc/sys/vm/overcommit_memory
fi

# Set open files
ulimit -n 65536

export HOSTNAME=$(hostname --fqdn)

# Run any migrations on startup
/app/bin/migrate

exec /app/bin/server

Partial Dockerfile

# Lots of other stuff
COPY --chmod=0755 --chown=nobody:root priv/scripts/docker_app_server_start /app/bin/run
CMD ["/app/bin/run"]