Ecto Auto Migrator - (thoughts appreciated)

Hi, Wonder if any experts could glance over my tiny library here:
GitHub - nippynetworks/ecto_auto_migrator

The substance is that sometimes there is a need to run migrations automatically at app startup and without an external mix migration step. Embedded would be one example, but I guess SASS products face the same.

The actual Ecto docs include a sample for creating a migration function, and elsewhere on this forum it recommends creating a simple genserver compatible module which can be added to the tree to run the migrations

All I have here is a simple skeleton around these two concepts as there was a bit of typing and a few constants to get correct. I will also need this in a couple of different projects.

Some extra lines I added were because I noticed that mix will of course start the app regularly, so will many build tools doing linting, etc, so I’ve wrapped the migrate function in a test which can be set in Config, which in turn I recommend it set via an env variable. This approximately gives the owner of the app the ability to run migrations in specific situations, eg deployed builds, but restrict it during development (where migrations might still be run manually)

My specific use case I need this migration step to pass no matter what (unattended embedded use case), so the migrate step is overrideable and in my case I replace it with some code that will blow away the DB (and recreate it) in the event of a migration failure (other options might be to restore a backup or ignore the migration, etc, depending on your situation)

Nothing terribly profound here, but it took me at least a couple of mins to get the architecture straight in my head, so hopefully this helps someone? Comments appreciated?

GitHub - nippynetworks/ecto_auto_migrator

6 Likes

OK, so I have made a few tiny tweaks to this upstream. Sure it’s nothing hugely inciteful, but perhaps it offers inspiration to others who want to attach the Ecto migrations to startup of the app, rather than via a strictly separate offline process.

In my case I’m running a headless router, so nobody to fix it if it goes wrong. What that looks like as an example of how to use this library is as follows:

In my config I have:

config :my_app, run_migrations: System.get_env("RUN_MIGRATIONS")

The app is then started with something like the following in dev, (set the env variable appropriately using your release process)
RUN_MIGRATIONS=1 iex -s mix

Then in my case I wanted to try the migrations, but if they fail then I wanted the app to start no matter what, so I blow away the DB. Figuring out how to do that for sqlite wasn’t totally trivial, so note the incantations below:

This module is started in my application tree soon after the Repo module

defmodule Database.Repo.Migrator do
  use Ecto.AutoMigrator
  require Logger

  @doc """
  Entry point

  Run DB migrations and try to ensure they succeed.
  Specifically we will delete all the DBs if migrations fail and try to re-run migrations from scratch
  """
  @impl true
  def migrate() do
    if run_migrations?() do
      load_app()

      try_migrations_1(repos())
    end

    :ok
  end

  # Run migrations, if they fail then blow away the DBs and retry the migrationss from scratch
  defp try_migrations_1(repos) do
    case try_migrations(repos) do
      :error ->
        Logger.critical("migration failure. Purging databases to attempt to continue")

        delete_databases(repos)

        # Retry from scratch and hope we can complete
        try_migrations_2(repos)

      :ok ->
        :ok
    end
  end

  # retry migrations second time
  defp try_migrations_2(repos) do
    case try_migrations(repos) do
      :error ->
        Logger.critical("migration retry failure. Continuing, but anticipate that app is unstable")
        :error

      :ok ->
        :ok
    end
  end

  # Delete all database files associated with all 'repos'
  # Currently assumes sqlite DBs
  defp delete_databases(repos) do
    for repo <- repos do
      repo.__adapter__.storage_down(repo.config)
      repo.__adapter__.storage_up(repo.config)
      # Purge all in use connections or we will still be using the old DB files
      repo.stop(5)
    end
  end

  # Try and run migrations, wrapping any exceptions and converting to :error/:ok result
  defp try_migrations(repos) do
    try do
      run_all_migrations(repos)
    rescue
      _ -> :error
    else
      _ -> :ok
    end
  end
end

I think this needs more attention. I decided to search the forum for other examples of this procedure but it appears nobody has really taken interest in this post or sharing other methods. I have been considering writing a library for this too but I’m short on time. I’m curious if you’d be interested in taking a look at the example I’ve put up here as it demonstrates the method I’ve been using which was created by Pleroma (https://pleroma.social)

It would be nice if there was a common library that everyone uses for this purpose.

1 Like

I think this didn’t get much traction because that’s a rather unusual approach to migrations. Depending on the environment/build env, the usual thing to do is:

Local development (:dev):
manually run

mix ecto.migrate

Local testing/CI (:test):
run migrations as part of the tests - the test task is aliased to

test: ["ecto.create --quiet", "ecto.migrate", "test"]

in mix.exs

Production (:prod):

If you’re using Docker, this could be:

CMD ["sh", "-c", "bin/app eval MyApp.Release.migrate && bin/app start"]

Usually there are healthchecks in place, so if the migrations fail, the deployment process should pick up on that and abort the deployment.

The case in the original post was about running the new version regardless if the migrations succeed or not, but I think this can be easily achieved with running the migrations as a separate command and just ignoring any errors.

2 Likes

Without disagreeing. My use case was an embedded device, so we will be booting the box after an upgrade when we realise we need to do a migration

I speculate that it would be helpful for SAAS kind of products. Eg I migrated away from gitlab to gitea because all that having to run migrations in the right order and remember all the commands was killing me. I just want the software to know what migrations to run and run them (sure if it goes wrong I will need to step in, but for many use cases short of clustered things, you will want the software to run the migrations I think?

Thanks for taking the time to reply! Appreciated

If you follow what @stefanchrobot wrote then running code automatically when starting the application vs. “from the outside” as shown is just a matter of calling MyApp.Release.migrate() within your application startup logic like e.g. MyApp.Application.start instead of via bin/app eval. In the end the created module is just a module with functions like any other. You can call them in a way which fits your project.

The benefit to doing migrations ouf of band is that you can uncouple software deploys from migrations, which is usually done for derisking deployment steps.

1 Like

Hmm, that doesn’t seem to be the “OTP Way” though?

How/Where do you run this code? How will you deal with failure? Retries? Ensuring that the rest of the app can start up around it? It may be an umbrella app with various dependencies that the OS author doesn’t know about ahead of time?

I’m struggling to see how you would implement your suggestion in various use cases. eg imagine I compile some app to be something like a desktop app for a user. This is a normal end user, who is expecting to double click and go. Now the app is upgraded (brew upgrade, etc) and they expect to be able to double click and go again. Where will this deploy logic fit?

Similarly, I have an embedded app, running on a box which is in the middle of a field that is a 10 hour journey away from any engineer who can visit it. It gets a remote upgrade (could be via a USB stick, remote net connection, whatever). It reboots and needs to deal with upgrading itself from whatever previous position it found itself in. For extra fun that app will be an umbrella app with various optional modules, eg we might have a choice of two control UIs, one is a basic web UI, simple DB used for storing config. The other is a sophisticated IOT device which needs a completely different DB to store all it’s config. We don’t necessarily know ahead of time which the box has installed, and of course the needs or quantity of DBs may change dynamically through time, so I can’t assume that the OS level knows enough to run the migrations

So yes, it’s all just code… So, agreed, it makes little practical difference if the OS pre-runs app.migrate, then boots the app, or if it just boots the app and the app runs app.migrate as part of startup. However, it may make a big speed difference on a little box which is upgraded infrequently, but needs to boot rapidly. So it’s impractical to run a separate migrate step on each boot. OK, so now we need a boot step which runs the migrations and in turn cascades those calls out to an as yet unknown number of apps within the umbrella, ie I need an app which is composable at runtime and doesn’t necessarily know about all the migrations of each of it’s subcomponents. So sure, I can run some code at app boot, but what if that code fails? How to re-run it, create a dependency tree, ie do all the OTP stuff. Well then it seems like we want an OTP module to be running our migratations and for that to be happening as part of the normal startup process - then we get all the goodness of our OTP system to manage failure, retries, etc.

Which brings us full circle to:

  • tada. Here’s some skeleton code to make it easy to build an OTP module that you can boot as part of your app startup to wrap your Repo.migration stuff and ensure that a) it’s run each startup and b) it’s within the OTP framework in order to be reliable

I realise that this forum has a large number of developers who are specifically paid to manage individual services and babysit their upgrade process. However, there is a large opportunity space for apps which are deployed without this babysitting and still need to be highly reliable, and manage their own upgrades without the benefit of a skilled developer babysitting it. If this isn’t you, then no problems, but I personally hate Gitlab and the immensely long and complicated Rails migration scripts that are needed. If the migration process is fixed then have the app run it as part of startup?!!

I claim that the rest of you guys are all wrong… :wink: I claim that (in many cases) the evolution is from being given some SQL to run as part of the upgrade, through to wrapping this in a migrator which is run as a separate “mix migrate” step, through to running this step automatically as part of the app’s normal (release) startup process (which of course should be done through OTP friendly mechanisms). In my case I add a few conditionals to this migration to avoid it being done repeatedly by automated tools within my editor (eg test watchers or linters, etc), I would usually avoid it being run automatically in dev env, however, I would probably have it run automatically in release/test environments.

My 2p

Re @feld’s comment. I do agree there are real needs for carefully managed migrations, where these need to be carefully controlled, run by an skilled engineer and failure is a showstopper.

However, I would claim that we should be working from the other end to get here. We should be
a) treating those migrations as code which is carefully tested (why should this be a third class citizen which requires a skilled user to carefully design a process around, separate to running the app? Generally I close my email program, upgrade it, open it again and expect it to take care of the details, not drop me to a shell and have me type stuff in…)

b) Once the migrations are a testable chunk of code which the app verifies are run and valid as part of startup, then we can work backwards, if required, and have separate pre-start process, which just part boots the app and checks that migrations are valid, or runs the migrations, or rolls them back or whatever.

This naturally leads to

a) thinking of migrations beyond just ecto databases, eg I might have other data which needs migrating, eg my “cubdb” database, or my crypto keyring or my exchange rates or my ssl trust chain, etc.

b) migrations can be “composable”. eg right now if I push some code to hex, and that code perhaps needs it’s own data store to store some state, then how do we pass control of that migration of state back to the top level of the app? Either the developer reads the README for the dependency and writes code, or the dependency just gets on with the job of migrating the data store when it starts up?

c) Similarly if I have an umbrella app, there may be optional pieces which may or may not have database dependencies. Now yes, “mix migrate” runs through all the sub apps and calls their mix commands, but if we were doing this as a release then we need more care as we need to write a function and have that function know about all the sub apps and whether those migrations succeed or not.

d) We can start to make these migrations OTP processes which can retry or handle failure or otherwise be smarter about success and failure. eg if we need to obtain a new SSL keychain, that might need downloading, it might need to handle failure, timeout, etc. Success or failure of that may or may not be a critical fail, but it might mean that further sub tasks are or are not run as a result and so on

e) I guess migrations don’t always only happen at boot? I’m thinking about say my currency rates or keychains, etc?

I don’t think there is a one size fits all solution to be found. However, my thoughts would be:

  • Make migrations a first class citizen of the app, test them
  • Make migrations an OTP process so that we can wrap infrastructure around them
  • Possibly we could make our apps have a skeleton which includes a “migrations” boot phase and encourage developers to put migrations into this area?
  • It would be helpful if there were some standard patterns and tooling around booting and running only the migrations phases of the app for use with “enterprise” apps where the migrations need to be carefully monitored and controlled for success.

Thoughts?

I would think a desktop app would have them run during the install and update, which would be decoupled from the starting of the app. You probably want to be able to uninstall an update.

Might be the case. Alternatively, you’re solving for a different use case.

Most of the web development needs zero-downtime deployments. This means that:

  • There is a time window during the deployment when the old version and the new version of the app are running at the same time,
  • The old version of the app must work with the migrated schema/data.

Let’s consider a common relational DB schema migration: adding a column to a table. Let’s say the migration failed.

Your approach seems to be suggesting that the app should still boot and run just fine. How am I supposed to write the app to use the new column since it might not be there if the migration failed?

I choose to avoid that problem (and the effort to somehow solve it) altogether by not booting the new version of the app, aborting the deployment process and letting the old version run. You might not have that convenience, but most of microservices do, so I see no reason not to take advantage of it.

There’s not much babysitting, since the migrations are run automatically as part of the deployment. Whether this is a separate process or happens in the app doesn’t really matter. The key is to abort the app boot on failed migrations.

So there’s a lot to uppack here.

First of all I’d like to address the fact that you’ve quite a lot of assumptions about the people you’re interacting here, which might not hold up once challenged. As it happens to be I’m working on an embedded system at work as well. To execute ecto migrations we do MyApp.Migrator.migrate in MyApp.Application.start and that works fine for us. Though I guess that’s besides the point.

I feel a lot of the points you brough up are “how do I deal with the complexity of migrating in my project” rather than “how do I execute migrations”. The answers you got are about the latter, which imo are a lot simpler to answer and also completely separate to potential answers to the former. Also it sounds like you’re working on a project, which is a lot more complex than what most people using ecto work on. So it’s to be expected for that situation to have less resources be available.

Depends on the failure. How likely is it that the failure is based on some temporary condition? If it is then retries might help. For us this is reboot on startup error. If not then retries are a waste of time. If a migration fails because the data in the db is in a unexpected state than it most likely needs human intervention to fix. No amount of preexisting code will help here. You’ll likely need some kind of circuit breaker to prevent the failure from cascading onto parts of the system you use for communication, given what OTP gives us is meant to keep “the whole” system running and stop if it can’t. Even for temp. problems backoff would be useful, which you don’t get with default OTP restarts.

Most often this is handled again by spliting up migrations from software deployment. In case of embedded software you might deploy a firmware, which just adds the migration to expose new/changed data, but not remove the old access point nor change the software itself. Then it doesn’t matter if migration fails, because nothing is depending on the new stuff yet. Once that’s successfully deployed you can later deploy just a software update, which makes use of the new data available. If your embedded device supports rollback you could also use that to potentially consolidate migration and software deployment with a fallback in case of errors. If you (or whomever handles your subcomponents) cannot “babysit” migrations that closely then you need to be more radical in not running things if they don’t work. Depending on the project downtime might be acceptable if things go wrong, if problems are at least eventually fixable.

If you’re deploying to embedded devices without knowing your dependencies you’re up for having a bad time no matter what you do. Your dependencies might include NIFs, which can crash the beam and there’s no way to catch that from within the beam.

Migrations are not executed on each run. Which migrations where successfully applied is stored in the db. So if there are no new migrations then it’s a single db query + a directory listing on the folder of migrations to check for that. You might be under constraints where those are already to slow, but I guess that’s not the usual case.

That’s already the case. While the generated setup shipped with phoenix does run migrations before tests start that’s just something to get people started easily. Given you’ve more complex requirements you can use a more elaborate setup, which tests migrations. Ecto.Migrator includes API to do that. For this one I actually think a library might be useful to integrate that a little more declaratively in tests. Ecto.Repo.put_dynamic_repo might even allow for running concurrently to sandboxed tests.

Running migrations on boot is the way to go for embedded devices, but it’s against common and adviced practice for deployments for general webservices. I don’t see this becoming an encouraged practise.

Tbh I fail to see how a release with some module to run migrations doesn’t essentially give you that. If you want to just run migrations do bin/app eval SomeModule.run_migrations. If you want to run migrations on boot you can put the same call in a boot phase or within MyApp.Application.start and it’ll be run on boot.

The complex part will be within SomeModule.run_migrations and depend on the project being built.

Tbh I fail to see how a release with some module to run migrations doesn’t essentially give you that. If you want to just run migrations do bin/app eval SomeModule.run_migrations. If you want to run migrations on boot you can put the same call in a boot phase or within MyApp.Application.start and it’ll be run on boot.

The complex part will be within SomeModule.run_migrations and depend on the project being built.

Can you flesh out this part please?

Near as I can see we are all proposing a core bit of code “SomeModule.run_migrations”. The discussion is going slightly sideways, but I’m essentially pondering who/what/when kicks this module off?

In mix land we can use a mix task. In release land we seem to need “eval” to kick it off. I’m basically proposing to upgrade it to a genserver and stick it in the app tree (which I don’t see as a million miles away from using an eval?)

Main issues I’m struggling with are how to "compose"migrations from multiple parts of an umbrella. In one of my projects I have optional apps included within an umbrella and these use a variety of DBs and hence multiple migrations, potentially scattered across the app and different for different builds. Presumably you are arguing in favour of using each individual Apps Application.start function?

Calling a fixed module from MyApp.Application.Start, or inserting the same code into the OTP tree seem very similar to me? The idea of using the OTP tree is that I could in theory decide to cascade failure or not… I don’t feel very strongly about this since it seems a bit nuanced for most use cases. However, given that if I don’t start up then I lose control over the whole box (not UI to have a user install a fixed firmware, no background service to allow an automated admin tunnel…), then it’s essential that I can try and ignore failures or take appropriate action (eg at least one repo is just logging data, so we can afford to blow that away to get back into a sane state.)

I’m also a bit unclear what happens if we have an umbrella app and one of the apps wants to fail it’s startup due to a failed migration? I guess it’s not so different if we fail in our OTP tree than the Application.start function? Will depend whether we set the app to :permanent? I guess I was thinking that putting it in the OTP tree gave me a bit more flexibility on when and how to fail (or ignore an error)

It seems to me that a web app might also want to struggle partly into life and offer some diagnostics to the user on how to rectify the failure? Dunno?

Note though that if YOU are automatically running migrations at app boot then we are already on the same page. What I’m basically advocating is that apps should move towards automating all (where practical) migration tasks and assisting the user in managing this process. Sure - offer me some advanced commands if it’s useful for me to run the migrations separately to booting the app, however, if migrations need running then don’t let me boot the app without running them! Fail, warn, disable functionality, whatever, but do everything possible to avoid involving me! (meaning for sure, don’t lose data, but if it’s case of running “Blah.run_migrations” then get on with it…!)

Thanks for taking the time to reply! Appreciated.

I think I’ve not set out my concerns which guide my subsequent thought process clearly enough (apologies)

  • I hate apps (looking at your rails), which have a deployment/migration document which has multiple “mix do_blah” alike steps that need absorbing

  • I don’t see that Elixir is particularly moving towards solving this. Deployment has appeared “problematic” until recently. Even putting your migrations into a Module.run_migrations module seems a like a bit of hidden knowledge (I think I found it buried in this forum as a tip, way down a thread…). Generally I see frequent assumptions that you have access to mix and some expert will run it and check for errors?

  • It feels to me that we the community could do more to encourage a clear entry point to Module.run_migrations and setting up some skeleton around this for deployment (ie mix won’t exist for lots of users at deploy time)

  • Booting up the app having forgotten to run the migrations seems like a horror that shouldn’t happen to me? How to prevent this and avoid corrupting data?

  • I (opinion) feel that lots of developers opt out of the difficult task of writing bullet proof migrations. If the migrations are run at the command line by some expert, then it’s much easier to punt corner cases to documentation and support forums. If in contrast I was told that my requirements are to run the migrations automatically at app startup I would emotionally be tackling the problem with a different level of rigour… (and testing!)

  • Some apps cannot be allowed not to start. Migration failure is something that needs to be managed, but can’t stop the app booting

  • I speculate that for some apps it would be nice if the migrations could be run in the background while the app is operating “normally”. eg some migrations might take hours or days and the app should try to operate during that process

  • In some of these cases migration failures might be soft retryable or mark some subset of the database needing attention

  • Migrations aren’t only about databases. There could be on disk data, config files, logs, non ecto databases, etc

  • Composable apps are sometimes tricky. If I wrote a Wordpress competitor, then I would likely need to support a bunch of plugins (yuck), which they themselves likely need to manage their data and version upgrades. Again this could consist of both traditional database and non database assets

  • External transactions. Groups of migrations may make a logical unit that can succeed or fail, eg upgrade all the WordExPress plugins or none, etc. Some migrations might be difficult/impossible to roll back (eg some of the non database stuff).

I think we might end up in an interesting place if we start thinking about the migrations being valuable code and data to manage, just like we do when the app is running. The current attitude of “squeezing it to the edge with a big bang change” seems at odds (to me), with the rich tools we have to manage and maintain the data after we “boot”? Do we end up in a different place if we think in terms of booting half the app before starting the migration process?

So I have a similarish desire in regard for migrations: startup the application and apply any outstanding migrations as needed in the startup routines. Naturally, there’s a bit more to it than that and I’m not in an embedded scenario, but there are reasons for this approach I believe are valid for wanting my application itself to govern the running of database updates rather than have that as a separate administrative task. So my solution: don’t use Ecto migrations for this purpose or try to bend it to my will.

My viewpoint as a still relative outsider to the world of Elixir is this: Ecto as a tool is designed to meet the needs of a general audience, but a certain kind of general audience. If you’re building a sort of typical web-app and your product is delivered via that web-app, where the database is mostly simpler persistence for a specific application, where you’re doing rigorous continuous integration & (probably) continuous deployment, you don’t have database specialists involved in development (so called “application DBAs”), and you have full control over both the development of the application and the operations of the application: my assessment of Ecto is that it’s most at home in this kind of environment. We can probably extend this a bit if we allow just the operational control by qualified staffing, where the application is brought in from elsewhere, like you might find in libraries or certain utility apps that you might add into your operation. (I’m probably speaking more about Ecto SQL throughout this post moreso than Ecto, but I’ll just keep saying Ecto for now, you know what I mean).

But leave the model that Ecto best supports and it quickly makes less sense to use and trying to make Ecto be all things to all people I think will ultimately diminish the value it does provide to its intended audience.

Watching the forums here and talk around the Web, you’d think that database access and Ecto are necessary partners, but they aren’t. Ecto is a very helpful library that will get you through a lot of use cases. But when it doesn’t hit the mark, it’s probably best to just leave Ecto and its opinions on the shelf and find something else (OK, there really isn’t a lot to find) or make something bespoke to deal with your specific needs… the complexity and size of Ecto is probably because the way it is because it’s trying to be a generalized solution for these kinds of applications.

I looked at the specific problems I’m trying to solve and decided to roll my own migrator (just finishing up that project). I don’t need many of the features that Ecto brings with it and Ecto doesn’t support a number of features I think are important to this specific application, and it’s not really that much code to make things work: it just needs to support my model for database updates well and handle the failure modes that can come out of that model in a sufficiently predicable manner. So in this regard, Postgrex is much more important to my application than Ecto SQL. But the requirements I’ve set for my application are niche compared to the common use cases for Elixir and I wouldn’t suggest anyone follow my model unless they have clear reasons to do something similar. Of course, there’s the query builders and the DSL around building out the schema with Ecto… but I don’t need that either. Plain old SQL is completely sufficient and I’m probably better served since the database does more in my application than in many web-apps.

Anyway… I guess I’m just trying to say is extending the target use cases in a well established something like Ecto isn’t necessarily bad, but it’s worth asking if you should rather than if you can and that’s basically the messaging that I’m seeing in this thread: many voices not sold on extending Ecto in this direction and I think they’re probably right.

Thanks,
Steve

P.S. Perhaps Ironically, I am using Ecto proper for it’s changesets and validations in dealing with internal validations, form processing, etc. I could gin up something for that, too, but for my use case it actually works well so far. I may end up going elsewhere in the end, but for right now those parts of Ecto are superior to anything I could build myself and will likely be more than good enough.

3 Likes

That’s exactly the complexity I was hinting at in my last post. I don’t think I have the one solution here given all approaches come with their individual tradeoffs. The more “dynamic” or unknown (as opposed to static) a system is the more tricky it becomes to handle its state. In the end you’ll need some place to know of all the subcomponents you have. When using the individual Application.start callbacks then this place would be the startup list of applications in your release. Generally I’d suggest having some central component in your system knowing about subcomponents, because then you are in control – instead of delegating to a piece of code you don’t directly control. With that you could have something like a MyCentralApp.run_migrations delegate to a known module in each subcomponents, which handles their individual migration needs. Something like

for sub <- fetch_subcomponent_roots() do
  Module.concat(sub, Migration).run_migration()
end

I’d strongly suggest to let each application deal with it’s own migration needs, ot

That’s imo completely unrelated to migrations, but to anything your dependencies do. I want to start that this is generally not something the beam has a simple solution to. If an application crashes (and it’s a permanent one) then the vm will shut down and is expected to come back online by means outside the vm. Also restarts are the means of recovery on the beam, but (especially without backoff) they can only fix a subset of possible errors in a system. The goal of tools provided by the beam and otp is to keep all the stuff running, which you tell it to keep running and it’ll give up if it cannot do so.

So to build a system, where partial availability is more important than complete availability you’ll need to build in the guard rails. Those could be circuit breakers, starting applications (see e.g. the shoehorn library) or processes with non :permanent restart types, using more elaborate restart mechanisms with backoffs, custom logic (see e.g. the parent library). You’ll essentially want to be able to shut off things, which continue to fail (causing partial downtime) in favor of keeping the ability to connect to the system from the outside. Doesn’t matter if a migration or other code is the cause of the errors.

This becomes even more tricky when you considere NIFs, which when they fail directly bring down the beam VM. Nothing application level can catch errors in NIFs. Also there are outside forces, which might stop your beam VM. E.g. when using linux the oom killer might stop the beam if you’re using to much memory.

While as you can see there are a lot of things to cover, this is not to say you cannot deal with those. But you’ll need to be diligent and careful, know your dependencies, know your failure cases, … The beam has many tools, which help build a system, which has the properties you need. You won’t be getting them automatically though.

2 Likes

The fine line which seems to getting agreement here is:

  • Migrations encompass more than just Ecto
  • Once you add subcomponents, it’s useful to disperse the migration process into the components

I think I’m seeing glimmers of agreement that: “having the app itself manage most migrations” is desired?

So, I hope the library as proposed gives some ideas on how that could be managed. I do see a missing piece with data migrations and how to handle in a robust form. I dislike the current tendency that it’s pushed to an upgrade readme (although I grant that the more “enterprise” or “clustered” the end result, the more appropriate this could be)

I think the final solution will be some guidance on good practice. Something like we have with deployments. Perhaps a startup phase in apps reserved to try and shoehorn in migrations, along with some useful lib functions?

Lets see

There already is detailed docs on how to use the migrator, and the examples shown applies to a release deployment. It already shows how to call migrations directly either through shell or a script through eval, and it can easily be placed in a startup shell script as well (or if you’re dockerizing your app, in the dockerfile’s CMD or manually through docker exec.).

The worst case possible of having migrations run before the deployment is if there are issues with your migration script (which may happen from time to time) due to data inconsistencies. You wouldn’t want a faulty migration to block your deployments.
Or another possible scenario would be if you need to downgrade your app’s version and roll back the migration manually, how would you do so if the migrations run automatically?
What if you’re doing hot code reloading for HA requirements?We’d need to decouple the migrations completely in that case.

Everyone’s deployment requirements differ, hence there isn’t a “best practice”.

You aren’t addressing the issue of the user running the app without running the migrations. This could equally cause trouble…?

Please read through the whole thread. I’m claiming that the old school desire to treat migrations as a bit of an afterthought is undesirable. Case in point, I upgraded my android phone yesterday, at no point was I dumped to a shell to manage the individual migrations of dozens of apps…

I think you’re missing the point that most people are not building software where some naive end user is doing the upgrades. I’d assume most people are writing software that they or their team deploys on servers and users just access a webpage.

If you’re trying to gather support for a library to meet your niche requirements maybe you could try being a bit nicer about it?

I have similar requirements to you as my app is run on clients’ hardware, often disconnected from the internet. I plan to run/rollback migrations as part of the installer/upgrade/uninstall.

I think they there’s already been much improvement in that respect, given that there is now releases and actual docs on how to do migrations, as compared to previously one had to work with distillery and complicate matters further (I know because I literally wrote an article on auto migrations with docker and distillery and startup hooks).

I don’t think that such deployment and migration complexity should be handled and solved by the ecto team, it would be above and beyond an average user’s deployment (1 app 1 release 1 migration needed).

Perhaps as a lib or cli tool, maybe.