Atomic deployment

Hello community,

There a lot of materials describing the strengths of the Elixir, but, unfortunately, deployment is not one of them yet. And that, in my opinion, neglects the overall value of every other positive thing about it, because of not being able to reliably operate in production is putting it in the “nice to play with” corner and holding back its wider adoption.

A couple months ago I wanted to ask if any of you were able to achieve atomic migrations with the native tools and now I believe none did, because of just couple versions ago sass didn’t emit a non-zero status on assets compilation failure https://github.com/brunch/brunch/issues/1671
and today I made a small effort to fix an alike situation in edeliver https://github.com/boldpoker/edeliver/pull/211

Edeliver is our central tool but known to be in need in lots of love :heart: The worst part about it is that it isn’t actively maintained and there are 15 PRs in the queue.

The point is that Edeliver along with @bitwalker projects are mission-critical projects for Elixir development and because of I’m not capable enough with bash-scripting to do it myself, I would like to raise community attention on that issue and have a coordinated effort towards bringing Edeliver to the state when we will have solid atomic deploys possibility, i.e. it will be possible to automatically rollback to the previous version on deploy failure.

That will be a huge win for everyone.

2 Likes

This part of Elixir is still a story in progress. I think though, this isn’t necessarily a specifically Elixir issue. While other languages and frameworks like Rails have tools like capistrano and chef/puppet, those can also be a nightmare to work with, and those are mostly considered mature tools.

I think the state of the idea behind codifying your deployment process rather than doing it by hand is still a developing community across all languages. Its my hope that technologies like Docker hope to simplify that, but I don’t believe there to be any one solution at the current moment. Theres just way too many things that can go wrong with lots of variables to handle in how different servers can be set up. Solving this well won’t just be a huge win for the Elixir community, but rather the developer community as a whole.

There a lot of materials describing the strengths of the Elixir, but, unfortunately, deployment is not one of them yet. And that, in my opinion, neglects the overall value of every other positive thing about it, because of not being able to reliably operate in production is putting it in the “nice to play with” corner and holding back its wider adoption.

I’m not sure I agree with this really. I definitely want to continue to improve the tooling, because it certainly has plenty of room for improvement, but I do think that where we are at is a lot better than many other platforms. I’m not going to comment on edeliver here because I don’t really know much about it beyond the basics - it doesn’t fit the various deployment models I’ve used since I started using Elixir professionally. That said, the underlying tech - OTP releases - is extremely solid, and I can’t think of any other platform that gives you the level of runtime operational tooling that they provide - it’s certainly way beyond “nice to play with”. It may sound like I’m patting myself on the back here, but exrm/distillery made it super easy to build releases in Elixir - in many cases all you need to do is run MIX_ENV=prod mix release and you have your deployment package - it’s not as simple as building static binaries, ala Go, but it’s still very painless.

Even in my current situation, where we are building releases, then composing an rpm package for installation, we are able to use pretty much the basic features of distillery + a single thin custom command to expose an interface for a shell script which acts as the CLI tool for interfacing with the application - and we do a lot of things with that.

Migrations, much like hot upgrades, are extremely tricky to get right - I would go so far as to say there is no general solution which will “solve” that. You may have multiple backend datastores which need schema updates, configuration changes, etc., which are non-trivial to roll-back (or perhaps even non-trivial to roll out in the first place). Your primary application may need complex appups to be hand written in order to do hot upgrades (if you are leveraging those in tandem with migrations). The story in Elixir around this is no different than any other language in my opinion, but you do have hooks at pretty much every stage of the deployment process to attempt to do these things with your own code.

There are definitely things that are left up to the consumer of tools like Distillery right now, and which might make it seem like there are warts, for example:

  • You need to be sure to set up your build environment so releases are built on the same OS/architecture as your prod environment. If you already have such infrastructure, this is basically a non-issue, but if you are trying to build on your laptop and deploy to DO or something, then there is some extra work involved (e.g. setting up Vagrant/Docker or something).
  • You have to understand the differences between Mix configuration and the static configuration used by releases (sys.config), and make sure you accommodate those differences. This is well documented, and there are a number of approaches, but it does mean things aren’t necessarily as easy as dropping in the tool and building a release.
  • There is no one-size-fits-all approach to doing migrations, because as I mentioned above, I don’t believe one is possible, and I prefer making it possible to layer on your own code to handle this and providing all the hooks necessary to do so. However there is official documentation on one way to handle them with Ecto, and a number of different approaches documented in the wild.

TL:DR - I don’t think the current state of deployment in the Elixir community is as dire as you make it out to be; but I do agree that there is plenty of room for improvement, more tooling which is more prescriptive which can make lives easier for those doing similar types of applications, and definitely more and better guides and documentation on how to get started and how to handle more advanced scenarios. I’d also like to add my voice to the plea for contributors to tools like edeliver (and selfishly, Distillery ;)), because contributions keep things moving along when maintainers like myself get swamped with their actual jobs, not to mention the myriad other things which distract from our open source work.

13 Likes

@bitwalker I wish edeliver offered same level of reliability that distillery does.

The point is having a rock-solid, automatable atomic deploys will be a huge improvement in the tooling.
Most Edeliver PRs are dealing with the BASH scripting and maybe having an expert in that matter will help project a lot.

1 Like

Which is why it is entirely useless in Windows networks like I still have to deal with at work…

In my opinion, the problem is complicated and significant enough that deserves a well-planned and coordinated effort to solve it. The shell scripting is a real pain-zone.

I told you folks https://dockyard.com/blog/2017/09/08/dockyard-hiring-to-solve-elixir-deployment

Now big boys in the hood will take a proper care of it

2 Likes

Atomic is a bit strong of a requirement. I am not aware of tools that give you that :slight_smile: if you are running a proper RDBMS (pg) you could do atomic migrations but beyond that …

Thanks! I just posted that in a thread here:

I reckon @bitwalker could be just the man for the job :003:

There are not much platforms that offer hot code swap too :wink:

I’ve been using https://nanobox.io/ for my personal project and it’s worked out really well. It hooks up to digital ocean or AWS and works basically like heroku’s command line, except one big difference: nanobox console web.main gets me a bash shell on the Phoenix box and then node-attach gets me a remote iex shell. They handled the SSL certs, rotate them every 90 days automatically, and have a really slick UI and handle (near) seamless deployments (app is down for a few seconds I’ve noticed but they’re working on it). Support on their slack channel is always top notch even at odd hours (not sure those guys sleep much).

I can’t recommend them enough, honestly it takes a huge load off me.