Help with deploying distillery upgrades

I’m having trouble getting distillery upgrade deploys to work: When I expand the 0.0.0 tarball in /app/run/ on a new machine, my “movies” app runs fine. If I put the 0.0.1 upgrade tarball in /app/run/releases/0.0.1/, then do “/app/run/bin/movies upgrade 0.0.1”, it tells me:

Release 0.0.1 not found, attempting to unpack releases/0.0.1/movies.tar.gz
Installed versions:
* 0.0.0    permanent
Unpack failed: {enoent,"/app/run/releases/0.0.1/movies.rel"}

I know the error message is coming from distillery’s release_utils when handling the error result of an rpc call into the node. This call is ultimately handled by unpack_release in Erlang’s standard library, which expects to see the .rel file at the root of the tarball, however, distillery puts that file in the releases subdirectory of the tarball with this Archiver code.

(Updated to add: distillery 1.0.0, Elixir 1.3.4, Erlang/OTP 19 [erts 8.2], building on and deploying to Ubuntu 16.10)

Any suggestions on what I’m doing wrong? (Thanks!)

Did you generate the 0.0.1 release with mix release --upgrade? If you didn’t pass --upgrade, then you will not be able to upgrade to that release.

1 Like

Thanks, @bitwalker – Yes, I generated both versions with MIX_ENV=prod mix release --env prod --upgrade (because there seems to be no harm, other than a warning, in generating the first version with --upgrade)

(Just went back and started from scratch, built 0.0.0 without --upgrade, then 0.0.1 with --upgrade – same result)

I’m still banging my head on this… I’ve created a new project from scratch, and written a little test script that builds and deploys version 0.0.1, then builds an upgrade 0.0.2 tarball and tries to deploy that. It’s here, with the test script and its output: https://github.com/bryanstearns/distillery_experiment

I was able to clone your project and run that script successfully (though I had to create prod.secret.exs). This was on my laptop, running OSX 10.11.6, Elixir 1.3.4, and Erlang 19.1. Could you open an issue here and we can dive deeper into this? I’ll have to try on my desktop, running Ubuntu 16.10, but won’t be able to do so until later this weekend.

Will do - thank you!

Actually, won’t… my test turned out to be invalid, and after cleaning up a problem with the test script (it did a poor job of killing off the daemon left around from the previous run) and an interaction with my shell environment, it seems to be working reliably.

I’m just gonna slowly step away from this for a bit :rolling_eyes:

@bitwalker, thanks for the help – happy holidays!

And as usual, stepping back from a problem is often the secret to finding a solution: while thinking about it in the shower this morning, it occured to me to wonder about filesystem permissions, and I was right!

Turns out that the cause of my original problem (in the first message in this thread) was that root was creating the releases/<version> directory and copying the upgrade tarball into it. It appears that unpack_release doesn’t notice that it can’t write to the target directory when it expands the tarball, but complains that it can’t find one of the files it should’ve written.

The app is running as the daemon user; chown -R daemon:daemon /app/run/$APP_NAME/releases/$APP_VERSION after copying the tarball makes the problem go away :smiley:

8 Likes

That’s a great find! Could you report that issue on the distillery tracker? I’ll see if we can try and do better with reporting that as an error.

5 Likes

Done! https://github.com/bitwalker/distillery/issues/173 – thanks again, @bitwalker