Is anyone doing CD with Hot Code Upgrade?

Hi folks,

I for one love Hot Code Upgrade. For one it’s an awesome capability that gives Elixir/Phoenix an unmatched advantage in a stateful world.
If you want to debate it - please also read and open a separate thread A Pipeline Made of Airbags. I’d love to keep this one focused on technical details.

With that out of the way - does anyone here do hot code upgrades with Phoenix?

What is your experience? What tools are you using?

Share your experience :heart: !

5 Likes

I believe @gregvaughn uses Hot Code Reloading with deploys. He has a short blog post about it, but doesn’t go into technical details:

1 Like

Hi. I guess I’m on the spot now :grin:

I have changed employers about a month ago. My current employer does not do hot deploys, but my former employer fo 4 years did. We were not technically doing Continuous Deployment. We had a script we had to have a human choose to execute that did our final build/deploy step. Sometimes we wanted to combine a few PRs into a single deployment and that was a simple way of dealing with it.

My former employer’s system was not UI-centric. We mostly did background tasks on behalf of our users. Our main UI was to let them see progress and occasionally manually update steps that needed human intervention. It was not a typical CRUD app. Hot deploys were great to keep all those background tasks uninterrupted.

We used distillery and edeliver. We had a build server that ran the same Ubuntu version as our production servers. Edeliver (which we had a love/hate relationship with) would build on that server and then deploy it to our production servers and issue the hot deploy commands (or full release depending on our manual params).

5 Likes

When COVID started to force countries into lock-down, our company was in a bad situation the current software would not have been able to handle, and we were forced to rush out a new product we’ve been working on for a while much faster than expected.

For the coming months, we had to hot-fix a lot of bugs, sometimes we had up to 20 deploys a day. And the software was in use, so any deploy with downtime in that frequency would become very annoying very fast.

We are deploying this project almost exclusively with upgrades via edeliver (have done so for ~a year now). Once we figured it out, it was actually a breeze and works very well.

Upgrades do have their pitfalls, and some things just need restarts - but we can live with these occasional downtimes (they’re not longer than a few seconds anyways now.)

I am sure there will be a lot of “should have used docker” kind of advice coming at me, but the reason why we did not use docker is simple and there was just no way around it: no one in our company knew how to set it up correctly, whereas we know how to run a vanilla Linux server… And we did definitely not have the capacity to figure out a new technology - so deploying upgrades was 100% the right decision, and I don’t regret it.
Like I said, it works and the few times we do have to do a hard restart are fine.

9 Likes

I’ve been using edeliver + distillery to hot upgrade our system for a few years now, works flawlessly. With recent versions there have been some issues updating the start_erl.data file, but I worked around that. Every now and then something more fundamental will change which would make a hot upgrade messy, and I have to do a full package deploy, but that doesn’t happen often.

My process mirrors that of @gregvaughn and the systems sound similar - most of the work my system does is data importing, processing, providing an API and integrating with third parties, and hot upgrades mean nothing skips a beat, no state is lost, etc. There’s no CD as such, just shell scripts that take care of building and deploying.

I know there are many ways to skin a cat, and most people seem to think there are better ways to accomplish the same thing, but I agree with @Ninigi - we’re a small business with no time or capacity to fiddle with and manage complex infrastructures, this works wonderfully for us - hot upgrades are magic.

5 Likes

I am sure there are people who would argue that docker is the easier technology to run - but that’s just a matter of perspective imo.

I have been running Ubuntu servers with different apps, or databases for a while - I have a decent grasp on how to use it. I even switched to a linux distro for my home-work environment, because why the heck not… So I feel like I at least understand the basics of what is going on, and usually at least have an idea what to google for. The one time we used docker on the other hand was a disaster, because no one knew what to do with it, and it really felt like all the knowledge of a basic server environment was completely wasted.

Long story short: I don’t think docker or any of the other technologies out there are bad. I don’t think “raw” edelivery and distillery deployments are necessarily better.

I personally feel better having a basic understanding of the technology I am using, and I totally understand what edeliver does (up to a certain point - I don’t think I could write it myself). If anybody has a different stack they know how to use, that’s great and I would not waste any breath trying to convince them to do it differently. At the time when this kind of stuff becomes really important, you should have enough budget to hire an engineer or two who specializes in this kind of stuff. Because this kind of stuff is hard man, there are just too many ways to do this kind of stuff.

3 Likes