Distillery/systemd reverting to last release instead of latest upgrade on app restart

Questions / Help
, , ,
1

So I’ve been writing a Phoenix app for a while now and have been deploying it with Distillery/Edeliver. I’ve set it up to use hot upgrades since it is likely to have multiple long-running requests going at any given point in time.

Everything is working fine with this setup so far, however, I’m having an issue with versions being reverted in the event of a crash. I’ve setup a systemd service but it keeps starting the latest release version instead of the latest upgrade version.

So, for example, I have a release version of 1.0.0 and I deploy a hot upgrade with version 1.0.1. If the app crashes for whatever reason or the system reboots, systemd kicks in and version 1.0.0 is started so the 1.0.1 changes are lost.

My systemd service file looks like the below (adapted from: https://hexdocs.pm/distillery/guides/systemd.html)

[Unit]
Description=Myapp App Daemon
After=network.target

[Service]
Type=simple
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu/myapp
ExecStart=/home/ubuntu/myapp/bin/myapp foreground
ExecStop=/home/ubuntu/myapp/bin/myapp stop
Restart=on-failure
RestartSec=5
Environment=LANG=en_US.UTF-8
EnvironmentFile=/home/ubuntu/config/myapp.env
SyslogIdentifier=myapp
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

I’m using a standard distillery configuration except with the following prod configuration (not sure if this is relevant):

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"..."
  set vm_args: "rel/vm.args"

  set config_providers: [
    {Distillery.Releases.Config.Providers.Elixir, ["${RELEASE_ROOT_DIR}/etc/config.exs"]}
  ]
  set overlays: [
    {:copy, "rel/config/config.exs", "etc/config.exs"}
  ]
end

Does anyone know how I could get distillery/systemd to start the latest upgrade instead of having it revert to the last release version?

Thanks in advance!

2

Yep, known issue! See: https://github.com/edeliver/edeliver/issues/314

3 Likes
3

Thanks so much @chasers! I completely missed that GitHub issue.

For anyone in a similar spot and looking for an alternative solution to the ones suggested in that issue, the way I fixed it for myself is to setup a cron script that copies and overwrites the start_erl.data file from ~/myapp/releases/ to ~/myapp/var/.

It’s not ideal but it seems to fix the issue until a fix is released

cp -rf /home/ubuntu/myapp/releases/start_erl.data /home/ubuntu/myapp/var/start_erl.data
2 Likes
4

A combo of the solution from the github issue thread and @dralith 's solution worked for me. This is the solution I used from the thread:

pre_erlang_generate_release() {
 status "Removing stale start_erl.data"
 __sync_remote "
   rm '~/myapp/var/start_erl.data'
 "
}

This worked when deploying new releases and upgrades, but on server reboots the app would still ignore the most recent upgrades back to the most recent release. I added @dralith 's cp code to an @reboot rule in crontab and now everything is hunky-dory until they release a fix.