What is your strategy for running one-off and automated tasks using releases (no mix)?

When deploying your Phoenix app via Elixir releases, how do you manage your database migrations? (and any other automated tasks). How do you run one-off tasks?

releases come with:

eval "EXPR"  Executes the given expression on a new, non-booted system
rpc "EXPR"   Executes the given expression remotely on the running system

I’m mostly fine with these if I’m migrating the database but ideally I’d love more structure than calling the code directly. I especially want more structure for one-off tasks. Shell scripts are an option but I’m not sure what the community recommends.

I came across this article, http://blog.firstiwaslike.com/elixir-deployments-with-distillery-running-ecto-migrations/ but it was written some time ago so am not sure if it still holds.

Any thoughts would be greatly appreciated. Thank you

Migration supervisor child.

children =
      [
        libcluster_child(),
        Sensetra.Endpoint,
        {Absinthe.Subscription, Sensetra.Endpoint},
        Sensetra.Repo,
        Sensetra.Repo.Migrator,
        # ... other children
        Sensetra.Ready
      ]

Endpoint is first so that there is a working /alive endpoint for liveliness checks. Then we boot the repo, run migrations, other children, and ultimately Sensetra.Ready runs to enable readiness checks.

Here is the migrator code:

defmodule Sensetra.Repo.Migrator do
  use GenServer
  require Logger

  def start_link(_) do
    GenServer.start_link(__MODULE__, [], [])
  end

  def init(_) do
    migrate!()
    {:ok, nil}
  end

  def migrate! do
    path = Application.app_dir(:sensetra, "priv/repo/migrations")

    Ecto.Migrator.run(Sensetra.Repo, path, :up, all: true)
  end
end

I wonder if ecto_sql should include a child for this. Then you could have:

{Ecto.Migrator, MyApp.Repo}

as a child.

EDIT: Created Redirecting to Google Groups

I should add that this style of running migrations assumes the following conventions about migrations:

  1. Database migrations are always backwards compatible. That is to say, after the migrations have run, the previous version of the code should still work. This is I think critical both because in rolling deploy scenarios you have two versions of your code running simultaneously, and also because if your new code has issues you make it much easier to roll back. You don’t need to worry about rolling back the database, just the code.
  2. New code is allowed to depend on the migrations. Because we can guarantee that the migrations happen at the right point in the supervision tree, code that runs later is allowed to use the new database changes. If the migrations won’t succeed, your new app won’t boot and won’t be routed any traffic.

When you need / want to do backwards incompatible database migrations you do so in two phases. You first roll out a version of the database schema that is backwards compatible with the old code, and contains new code that no longer depends on the stuff you want to get rid of. Then you do a second deployment after that has succeeded where you eliminate the stuff you are getting rid of, because this is now a backwards compatible change.

10 Likes

Thank you for the detailed explanation. Sorry for the silly question… if i have one app running and one (small–ish) db, do I need a GenServer to run the db migrations?

The main reason I think it’s still worthwhile: You can’t forget to do them. Plus, there’s no moment of time where the new code is running and the migrations haven’t run.

When you say one app running do you mean just one instance of your app?

1 Like

Yes, I mean one instance.

Thank you so much for the help!

Is this the only way? During development I coded an “import” mix task thinking that it would be available in production. But it’s an “one-shot” task. Any idea how to run it from the release ?

I’m not sure exactly how releases work, but perhaps you can just call the run/1 function of your task?

We use this approach: Custom Commands - Distillery Documentation

HTH!

mmm is this working with the new mix release ( mix release — Mix v1.16.0 ) task of the latest Elixir? The problem I have with using the eval subcommand is that I need to pass some parameters to the task…

No, it’s part of Distillery.

I use this as well. One thing that I like about it is that migration happens before any pods shut down. This means that if the migration failed for some reason (it can happen), we can run it again. No pods would get taken offline throughout that process, so availability remains at 100%.

I also like Ben’s approach, although there’s different tradeoffs.

This is way late, but I want to point out that no pods get taken offline in my version either. The migrator runs before the readiness checks are set to true, so K8s won’t start taking down any of the previous version pods until the migrations successfully run and the new versions mark themselves as ready.

1 Like

Ah, makes sense! Thanks for sharing that bit.

1 Like

Thanks Ben!

Sorry to raise this thread from the dead, but current versions of Phoenix add the Phoenix.Ecto.CheckRepoStatus plug to your default Endpoint pipeline. This raises a Phoenix.Ecto.PendingMigrationError exception in development when trying to hit your healthcheck URL while your migrations are still running.

To disable this, just remove the Phoenix.Ecto.CheckRepoStatus plug from your endpoint.ex:

if code_reloading? do
  plug Phoenix.CodeReloader
  # plug Phoenix.Ecto.CheckRepoStatus, otp_app: :my_app
end
1 Like