Is nerves using distillery release hooks?

Nerves strips out any runtime components of Distillery. Check this post i wrote a while back to run Ecto migrations on Nerves devices.

Nerves does not support using distillery release hooks. This is because Nerves uses a program called erlinit to boot erlang as early and as safely as possible. Writing embedded systems using Nerves means that your application code will live mostly in Elixir / Erlang. This means that its really important that the VM comes up and stays running to prevent bricking the device. To make the boot process more reliable we created shoehorn, an OTP app that is included in Nerves projects by default.

Shoehorn provides two major advantages.

1. If your application or any of its dependencies fail to start on boot, the VM can continue to run.
2. Applications can be put into a priority run level to be initialized before any other dependencies and your app.

The second point is the one where you can perform actions similar to that of Distillery release hooks. Using Blinky as an example, we can see how the initial config.exs file is generated:

config :shoehorn,
  init: [:nerves_runtime, :nerves_init_gadget],
  app: Mix.Project.config()[:app]

The option :init takes a list of OTP applications, and MFA’s to execute before the main application start. This is an ordered list. the default [:nerves_runtime, :nerves_init_gadget] means that :nerves_runtime and its deps will start first, followed by :nerves_init_gadget and its deps, followed by the main app declared in the key :app. You could add other OTP application names to this list or MFA’s such as init: [{IO, :puts, ["init_1"]}]. Shoehorn has an example app that performs MFA’s on init here

Thanks a lot for your detailed answer @mobileoverlord!

What I’m trying to do is to run migrations for the Phoenix UI. I’ve added the migration release task to shoehorn but I get this error:

** (RuntimeError) connect raised KeyError exception: key :database not found.The exception details are hidden, as they may contain sensitive data such as database credentials. You may set :show_sensitive_data_on_connection_error to true if you wish to see all of the details
    (elixir) lib/keyword.ex:389: Keyword.fetch!/2
    (postgrex) lib/postgrex/protocol.ex:90: Postgrex.Protocol.connect/1
    (db_connection) lib/db_connection/connection.ex:66: DBConnection.Connection.connect/2
    (connection) lib/connection.ex:622: Connection.enter_connect/5
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: nil
State: Postgrex.Protocol
[error] GenServer #PID<0.649.0> terminating

I guess this is because the Ui app has not started yet.

I start the apps before running the migrations, this is the script based on the distillery recommendations:

defmodule Ui.ReleaseTasks do # More info on: https://embedded-elixir.com/post/2017-09-22-using-ecto-and-sqlite3-with-nerves/ https://github.com/bitwalker/distillery/blob/master/docs/guides/running_migrations.md 
  @start_apps [
    :crypto,
    :ssl,
    :postgrex,
    :ecto,
    :logger,
    # If using Ecto 3.0 or higher
    :ecto_sql
  ]

  @repos Application.get_env(:ui, :ecto_repos, [])

  def migrate() do
    repos_pids = start_services()

    run_migrations()

    stop_services()
  end

  def seed(_argv) do
    start_services()

    run_migrations()

    run_seeds()

    # Stop servicej
    stop_services()
  end

  defp start_services do
    IO.puts("Starting dependencies..")
    # Start apps necessary for executing migrations
    Enum.each(@start_apps, &Application.ensure_all_started/1)

    # Start the Repo(s) for app
    IO.puts("Starting repos..")

    # Switch pool_size to 2 for ecto > 3.0
    Enum.map(@repos, & &1.start_link(pool_size: 2))
  end

  defp stop_services do
    IO.puts("Success!")
    :init.stop()
    IO.puts("Stopped!")
  end

  defp run_migrations do
    Enum.each(@repos, &run_migrations_for/1)
  end

  defp run_migrations_for(repo) do
    app = Keyword.get(repo.config(), :otp_app)
    IO.puts("Running migrations for #{app}")
    migrations_path = priv_path_for(repo, "migrations")
    Ecto.Migrator.run(repo, migrations_path, :up, all: true)
  end

  defp run_seeds do
    Enum.each(@repos, &run_seeds_for/1)
  end

  defp run_seeds_for(repo) do
    # Run the seed script if it exists
    seed_script = priv_path_for(repo, "seeds.exs")

    if File.exists?(seed_script) do
      IO.puts("Running seed script..")
      Code.eval_file(seed_script)
    end
  end

  defp priv_path_for(repo, filename) do
    app = Keyword.get(repo.config(), :otp_app)

    repo_underscore =
      repo
      |> Module.split()
      |> List.last()
      |> Macro.underscore()

    priv_dir = "#{:code.priv_dir(app)}"

    Path.join([priv_dir, repo_underscore, filename])
  end
end

I think it fails because the Ui app has not started yet.

I’m not 100% sure about this, but I think the Ui is started with the app because it’s a dependency, but could I move the Ui start to soehorn instead of being done automatically? After that I could put the migrations script and remove the start_services and stop_services functions.

Does it make sense? Could it fail because starting up the db takes time and the migrations will run immediately?

@ConnorRigby the code I use is pretty similar to yours. Start the apps, migrate and stop.

I added to the Ui application.ex

  def start(_type, _args) do
    import Supervisor.Spec

    # Migrate on startup
    Ui.ReleaseTasks.migrate()
...

but it doesn’t work either. The migrations work but after that, the Ui won’t start:

00:00:16.706 [error] Supervisor 'Elixir.Ui.Supervisor' had child 'Elixir.Ui.Repo' started with 'Elixir.Ui.Repo':start_link() at undefined exit with reason {already_started,<0.720.0>} in context start_error
00:00:16.706 [error] CRASH REPORT Process <0.718.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,'Elixir.Ui.Repo',{already_started,<0.720.0>}}},{'Elixir.Ui.Application',start,[normal,[]]}} in application_master:init/4 line 138

The repos were already started, I tried to stop them but somehow I still get this message.

So the reason that the shoehorn method won’t work seems to be because of a configuration error. I can’t say i’ve ever used this method so i’ll let Justin chime in on that.

I still use ecto 2 so it’s possible something changed in the migration code, but in my application after running migrations, the repo is stopped. Which adapter are you using?

I used your code in the past and worked fine, I moved to PostgreSQL (the current adapter) and still worked fine. This was some months ago. After this I worked only in the UI, and in the way I upgraded versions. Everything worked for Phoenix, but last week when I tested the Ui in nerves, I got some issues. This is the one that is blocking the device to work.

are you using Ecto2 or Ecto3? as a quick hack you could essentially stop all Ecto/Repo processes before exiting the Ui.ReleaseTasks.migrate() code

I already did that with:

repos_pids = start_services()

IO.puts("Stopping repos...")
Enum.each(repos_pids, fn {:ok, pid} -> GenServer.stop(pid, :normal) end)
IO.puts("Repos stopped")

And didn’t work.

I’m getting some results now.

I tried to start the ui and after that run the migrations adding at the end of shoehorn:

:ui,
{Ui.ReleaseTasks, :migrate, []}

I removed the start_services and stop_services because the ui was supposed to start it, but migrations didn’t run.

Until I found out that I had a genserver for an AMQP service with RabbitMQ that had the wrong credentials, so it didn’t connect to RabbitMQ and the Ui was never fully started so migrations never run.

I just commented that genserver and migrations seems to run, but I’m just testing yet.

If I make this system work again in the RPI3, with custom buildroot with PostgreSQL, a nice Ui with Elm and websockets, realtime events using RabbitMQ, I’ll be very very excited

btw @ConnorRigby I forgot to mention I’m using Ecto3

It works!

After fixing the issue with the AMQP genserver adding this to shoehorn worked:

:ui,
{Ui.ReleaseTasks, :migrate, []}