Phoenix live/in-place updating strategies

I would like to learn what my options are for updating a live Phoenix app with minimal downtime. There are probably guides on this topic, but since this is still early in the life of Phoenix (relatively speaking), I don’t want to spend time chasing old leads if they aren’t currently useful.

Can someone point me to a recent guide or perhaps describe here a strategy they use?

My challenge is that I have a live app that always has a few users, day or night. If I could replace little pieces (such as a controller+template) individually, then I could do in-place updates with practically no downtime.

Instead, I find myself rebuilding the whole app - which takes 30-90 seconds, enabling a 503 page, restarting cowboy, waiting what seems like forever for it to kill old processes and start new ones, and then remove my 503 page. While the building of the app isn’t visible to users, the app restart is.

I realize this is considered unprofessional, but I miss the days when I could replace a JSP or PHP page and it would recompile on the fly with no downtime. :slight_smile:

Sounds like you are not running releases and instead are running inline? You should always make a release via distillery or so when running production. Releases enable hot-code swapping, rolling upgrades (basically turning it off and back on, if you do it via different ports with nginx as a reverse proxy then no down time with this method, this is what I use), among others.

And what if your database changes shape? Or cached data changes? Or a request coming in when only ‘some’ files have updated during an upload? Or etc… They may have been easy but they had no way to handle permanent storage upgrading without breaking things for at least a period of time too. :wink:

/me did a LOT of PHP programming due to a job over a decade ago

How do you manage that? Do you manually set the port each time for the release with the port being in expected port range based off your nginx config?

Eh my way is hacky but it works. I have a main process running as a systemd service, my release script just builds a new release, starts up the server on another port, waits a couple seconds to let it spool up, tells nginx to swap to the new port and has nginx reload its configuration nginx -s reload, waits 15 seconds (I figured that is enough time to let any current connection die), stops the service, updates the service code, starts the service, waits a second to spool up, tells nginx to swap back and reloads its configuration, waits 15 seconds and shuts down the temporary one.

A better way that I keep intending to swap to is just to have nginx try both ports and just swap back and forth between them on updates.

Channels will die on web pages but they reconnect almost instantly so that’s no big deal, otherwise no one has noticed a thing at work. ^.^

Ahh you’re right! I forgot that I made an effort with two different published methods, one being distillery, to “do the right thing”, but ultimately could not make it work. So I gave up and setup supervisor to run the app in foreground.

I’ll revisit distillery and see if I can make it work this time.

I have setup, built, configured, and integrated so many different systems over time, I don’t think my failure with distillery was because I was a complete noob. But by the same token, when my brunch suddenly stopped working, and solving it resulted in manually installing ansicolors and a handful of other things, it really makes me doubt the completeness of “standard” solutions. Ultimately I have come to find that “whatever makes the damned thing run stably” is what I have to do. /rant

Right, when there are database changes, there’s a limit to how fluidly a hand-patched upgrade can work. But I write shell scripts, create new table names, new schemas if necessary, and live with temporary duplication until I can retire old data stores.

This port swapping idea is great! I could automate that easily. Thanks for the idea.

Distillery only recently’ish become complete in my opinion, you may have had issues prior to that. If you have any issues now please post them here. :slight_smile:

I should probably package up all my hacks and stuff into a deployment library sometime for people using systemd/nginx (and those could probably be fairly trivially made generic). ^.^;