I have made a data migration module following this article which allows me to run custom modules responsible for modifying the data that already lives in production. These might be as simple as running an update query on a table, or they might update data in an external system. Data migrations essentially mimics ecto migrations by having a folder full of migration modules, and it uses a table to track which data migration modules have already been run.
The goal is to run this data migration module when we deploy our release so that we can keep our data consistent without any downtime or manual intervention.
The problem is that I’m very new to elixir releases and I’m very lost on how to go about running this data migration module (that will in turn run all our unrun data migrations). How do I go about running a module when we deploy a release? Is there a best practice for this?
I’ve had a good look over a lot of the documentation (including that article). That article simply says to run _build/prod/rel/my_app/bin/my_app eval "MyApp.Release.migrate" which is fine, but it doesn’t say how to actually run it in prod without manually sshing into the machine and running that command.
How do I go about including something along those lines in a config file or some file that Elixir will run automatically?
defmodule Mango.Release do
@app :mango
def migrate do
for repo <- repos() do
{:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
end
end
def rollback(version) do
for repo <- repos() do
{:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
end
end
defp repos do
Application.load(@app)
Application.fetch_env!(@app, :ecto_repos)
end
end
so you can do something like that
ssh yourprod cd /path_to_prod_dir && bash ./migration_script
Hmmm but how are you running that bash file?? I’m currently deploying my app by using Google Cloud Platform App Engine Flex with a ready made yaml file by google which I’m not sure how to modify (Link). It would be great if there is a way to run this bash script from elixir somehow
If you want to run it from elixir, create migration module that follow elixir application loader (like genserver or so) put it in the application file and add it to the startup applications
def start(_type, _args) do
children = [
# Start the Ecto repository
Justice.Repo,
# Start the Telemetry supervisor
JusticeWeb.Telemetry,
# Start the PubSub system
{Phoenix.PubSub, name: Justice.PubSub},
Justice.Migrate, # <==============================================
# Start the Endpoint (http/https)
JusticeWeb.Endpoint
# Start a worker by calling: Justice.Worker.start_link(arg)
# {Justice.Worker, arg}
]
Interesting, adding it to startup applications seems reasonable. Is there any best practice around this topic? For example, is it bad to run a migration application every time our main application is started? Is this something people generally just ssh into and run the command as a once off thing?
I figured Elixir releases would have some best practices around running scripts or modules when you redeploy (e.g. you could add a module to some kind of hook) but there doesn’t seem to be anything of the sort.
I’m not actually running migrations, I’m running custom modules that look like migrations (they are only run once but they let me do any custom logic that I need to do to keep data consistent after an ecto migration). Keen to hear if anyone else has a different way of tackling this problem aside from SSH or adding a module to the startup applications. This module just has to run once on deployment and then can stop.
If you use mix releases, you can add your custom start-up steps to env.sh just like this:
case $RELEASE_COMMAND in
start*|daemon*)
# Put here commands that should be used on every start
;;
*)
;;
esac
So they will be run every time you deploy a new version of your application (I expect that you don’t use hot code reloading and every deploy means that you have to restart your software).
