Ecto Auto Migrator - (thoughts appreciated)

Hi, Wonder if any experts could glance over my tiny library here:
GitHub - nippynetworks/ecto_auto_migrator

The substance is that sometimes there is a need to run migrations automatically at app startup and without an external mix migration step. Embedded would be one example, but I guess SASS products face the same.

The actual Ecto docs include a sample for creating a migration function, and elsewhere on this forum it recommends creating a simple genserver compatible module which can be added to the tree to run the migrations

All I have here is a simple skeleton around these two concepts as there was a bit of typing and a few constants to get correct. I will also need this in a couple of different projects.

Some extra lines I added were because I noticed that mix will of course start the app regularly, so will many build tools doing linting, etc, so I’ve wrapped the migrate function in a test which can be set in Config, which in turn I recommend it set via an env variable. This approximately gives the owner of the app the ability to run migrations in specific situations, eg deployed builds, but restrict it during development (where migrations might still be run manually)

My specific use case I need this migration step to pass no matter what (unattended embedded use case), so the migrate step is overrideable and in my case I replace it with some code that will blow away the DB (and recreate it) in the event of a migration failure (other options might be to restore a backup or ignore the migration, etc, depending on your situation)

Nothing terribly profound here, but it took me at least a couple of mins to get the architecture straight in my head, so hopefully this helps someone? Comments appreciated?

GitHub - nippynetworks/ecto_auto_migrator

6 Likes

OK, so I have made a few tiny tweaks to this upstream. Sure it’s nothing hugely inciteful, but perhaps it offers inspiration to others who want to attach the Ecto migrations to startup of the app, rather than via a strictly separate offline process.

In my case I’m running a headless router, so nobody to fix it if it goes wrong. What that looks like as an example of how to use this library is as follows:

In my config I have:

config :my_app, run_migrations: System.get_env("RUN_MIGRATIONS")

The app is then started with something like the following in dev, (set the env variable appropriately using your release process)
RUN_MIGRATIONS=1 iex -s mix

Then in my case I wanted to try the migrations, but if they fail then I wanted the app to start no matter what, so I blow away the DB. Figuring out how to do that for sqlite wasn’t totally trivial, so note the incantations below:

This module is started in my application tree soon after the Repo module

defmodule Database.Repo.Migrator do
  use Ecto.AutoMigrator
  require Logger

  @doc """
  Entry point

  Run DB migrations and try to ensure they succeed.
  Specifically we will delete all the DBs if migrations fail and try to re-run migrations from scratch
  """
  @impl true
  def migrate() do
    if run_migrations?() do
      load_app()

      try_migrations_1(repos())
    end

    :ok
  end

  # Run migrations, if they fail then blow away the DBs and retry the migrationss from scratch
  defp try_migrations_1(repos) do
    case try_migrations(repos) do
      :error ->
        Logger.critical("migration failure. Purging databases to attempt to continue")

        delete_databases(repos)

        # Retry from scratch and hope we can complete
        try_migrations_2(repos)

      :ok ->
        :ok
    end
  end

  # retry migrations second time
  defp try_migrations_2(repos) do
    case try_migrations(repos) do
      :error ->
        Logger.critical("migration retry failure. Continuing, but anticipate that app is unstable")
        :error

      :ok ->
        :ok
    end
  end

  # Delete all database files associated with all 'repos'
  # Currently assumes sqlite DBs
  defp delete_databases(repos) do
    for repo <- repos do
      repo.__adapter__.storage_down(repo.config)
      repo.__adapter__.storage_up(repo.config)
      # Purge all in use connections or we will still be using the old DB files
      repo.stop(5)
    end
  end

  # Try and run migrations, wrapping any exceptions and converting to :error/:ok result
  defp try_migrations(repos) do
    try do
      run_all_migrations(repos)
    rescue
      _ -> :error
    else
      _ -> :ok
    end
  end
end

I think this needs more attention. I decided to search the forum for other examples of this procedure but it appears nobody has really taken interest in this post or sharing other methods. I have been considering writing a library for this too but I’m short on time. I’m curious if you’d be interested in taking a look at the example I’ve put up here as it demonstrates the method I’ve been using which was created by Pleroma (https://pleroma.social)

It would be nice if there was a common library that everyone uses for this purpose.

1 Like

I think this didn’t get much traction because that’s a rather unusual approach to migrations. Depending on the environment/build env, the usual thing to do is:

Local development (:dev):
manually run

mix ecto.migrate

Local testing/CI (:test):
run migrations as part of the tests - the test task is aliased to

test: ["ecto.create --quiet", "ecto.migrate", "test"]

in mix.exs

Production (:prod):

If you’re using Docker, this could be:

CMD ["sh", "-c", "bin/app eval MyApp.Release.migrate && bin/app start"]

Usually there are healthchecks in place, so if the migrations fail, the deployment process should pick up on that and abort the deployment.

The case in the original post was about running the new version regardless if the migrations succeed or not, but I think this can be easily achieved with running the migrations as a separate command and just ignoring any errors.

1 Like