How to run a custom module when starting a release in production?

Hello!

I have made a data migration module following this article which allows me to run custom modules responsible for modifying the data that already lives in production. These might be as simple as running an update query on a table, or they might update data in an external system. Data migrations essentially mimics ecto migrations by having a folder full of migration modules, and it uses a table to track which data migration modules have already been run.

The goal is to run this data migration module when we deploy our release so that we can keep our data consistent without any downtime or manual intervention.

The problem is that I’m very new to elixir releases and I’m very lost on how to go about running this data migration module (that will in turn run all our unrun data migrations). How do I go about running a module when we deploy a release? Is there a best practice for this?

Thanks in advance for the help!

You can look here how to do it

I’ve had a good look over a lot of the documentation (including that article). That article simply says to run _build/prod/rel/my_app/bin/my_app eval "MyApp.Release.migrate" which is fine, but it doesn’t say how to actually run it in prod without manually sshing into the machine and running that command.

How do I go about including something along those lines in a config file or some file that Elixir will run automatically?

I am using it in production
I created a small bash file

mango is my application name

export MIX_ENV=prod
export SECRET_KEY_BASE=0a7thhtpjnVq7Je2IW7zyE6v2xs/zFZUIGNz+1h9XEGJGkZuL1nFR32dKO7Dhnw/
export DATABASE_URL=postgres://pguser:pgpass@localhost:5432/mango_prod
export PORT=9090
_build/prod/rel/mango/bin/mango eval "Mango.Release.migrate"

and here is my migration code

defmodule Mango.Release do
  @app :mango

  def migrate do
    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(version) do
    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
    end
  end

  defp repos do
    Application.load(@app)
    Application.fetch_env!(@app, :ecto_repos)
  end
end

so you can do something like that

ssh yourprod cd /path_to_prod_dir && bash ./migration_script

Hmmm but how are you running that bash file?? I’m currently deploying my app by using Google Cloud Platform App Engine Flex with a ready made yaml file by google which I’m not sure how to modify (Link). It would be great if there is a way to run this bash script from elixir somehow

I am not familiar with google cloud but running remote command from ssh is easy

If you want to run it from elixir, create migration module that follow elixir application loader (like genserver or so) put it in the application file and add it to the startup applications

 def start(_type, _args) do
    children = [
      # Start the Ecto repository
      Justice.Repo,
      # Start the Telemetry supervisor
      JusticeWeb.Telemetry,
      # Start the PubSub system
      {Phoenix.PubSub, name: Justice.PubSub},
      Justice.Migrate,  # <==============================================
      # Start the Endpoint (http/https)
      JusticeWeb.Endpoint
      # Start a worker by calling: Justice.Worker.start_link(arg)
      # {Justice.Worker, arg}
    ]

Interesting, adding it to startup applications seems reasonable. Is there any best practice around this topic? For example, is it bad to run a migration application every time our main application is started? Is this something people generally just ssh into and run the command as a once off thing?

I figured Elixir releases would have some best practices around running scripts or modules when you redeploy (e.g. you could add a module to some kind of hook) but there doesn’t seem to be anything of the sort.

Migration will run only once and if there is no migration to perform, it wont run it
I never use migration this way so you have to find out yourself :slight_smile:

Thanks for the help! :slight_smile:

I’m not actually running migrations, I’m running custom modules that look like migrations (they are only run once but they let me do any custom logic that I need to do to keep data consistent after an ecto migration). Keen to hear if anyone else has a different way of tackling this problem aside from SSH or adding a module to the startup applications. This module just has to run once on deployment and then can stop.

If you use mix releases, you can add your custom start-up steps to env.sh just like this:

case $RELEASE_COMMAND in
  start*|daemon*)
    # Put here commands that should be used on every start
    ;;
  *)
    ;;
esac

So they will be run every time you deploy a new version of your application (I expect that you don’t use hot code reloading and every deploy means that you have to restart your software).