Hi, I need a way to handle data migrations in my application. I found an article by @wojtekmach about manual migrations: Automatic and manual Ecto migrations - Dashbit Blog. That was very informative.

What the blogpost suggests is to have a new kind of migrations: manual migrations. But they still use the same rules as the usual (automatic) migrations: data is changed via SQL.

The challenge I’m facing right now is that I need to do a bit more involved data migrations. Schema-less queries is an option, but I’m not sure it will cover all the use cases. Some of the data in the database is encrypted at rest and can only be read/written by the application (it can be copied though). The other thing is that some of those migrations are more like backfills and they require the application logic which means they need to rely on the structs which has already bitten us and caused a migration to fail.

Any thoughts on how to approach this? I’ve started going the way of adding a new release task that runs the data migrations, but then I loose the benefit of the Ecto migrations keeping track of what has already been run. Writing idempotent data migrations is a solution but there’s no easy way to enforce that and it sounds like a decent potential for really bad errors.

My other concern about migrations is - why are they done as scripts and are not compiled into the application? The concern is coming from my thought of using structs in the migrations: the script may become broken over time (this has already happened at least once).