I’ve had a look at a variety of sources, and this helpful response by fnux to What kind of server/infrastructure are your Elixir apps hosted on?. I regularly attend DevOps meetups too, enough to know that the big guys like banks and flight scanners have difficulties.
I’m confused as to the “right” way to go for DevOps CI/CD/CD best practice. That “right” is probably applicable to many project use cases in the Elixir community, if not further afield.
Focusing on Elixir/Phoenix, I need to set up a CI/CD/CD pipeline to manage many developers using Phoenix IDE then: GitHub/GitLab, CI build to include Gulp for Bootstrap and other assets, testing, Continuous Development, Continuous Distribution, monitoring. How/where does Distillery fit into that pipeline (if it does)?
I’m seeking to avoid cloud services, and especially AWS. Instead I’m using Debian Jessie/Stretch for Development/Production, Staging, Distribution and monitoring with several servers, bare metal and VMs, in different parts of the world. When established, I’m keen to release this project pipeline, with a large part of our code, for open source with free video training references. However, the project will be closed source until ready for release, so ineligible for OS freebies.
Jenkins, although popular, seems old and faded compared to “modern” offerings like CircleCI. I want to avoid facilities like CircleCI that cost money, simply to point lone developers and early starters to something that doesn’t quickly bog them down in costs. I’m guessing there must be something out there that’s modern and effective, but there’s so many CI/CD tools these days it’s hard to choose.
Apologies again for long post. Best practice/solutions?
How/where does Distillery fit into that pipeline (if it does)?
Distillery is a release tool. It builds a deployable artifact. You have two options with Distillery:
Build on the same architecture and operating system as the one you deploy on. This would mean compiling on Debian Jessie (or Stretch; I suggest you pick one or the other) running the same architecture as your deploy target. The result it a tarball that contains a runnable instance of your application. It already includes Erlang, so there’s no need to provision your deploy targets.
Exclude ERTS (Erlang Runtime System) from your release. This allows you to build on any architecture and operating system and results in a tarball containing your application. Since it does not include ERTS, you’ll need to provision your system with Erlang. I would recommend installing it using a package from Erlang Solutions.
How you get the deployable artifact from your build server to your deploy targets is not handled by Distillery. Everyone’s needs are different. Some people use tools that SCP the tarballs to each server and install the service. Some people copy the release into a Docker image which is then usable in any number of orchestration facilities.
There are caveats to 2: Elixir/Erlang allow you to provide functionality implemented in C (called native implemented functions or NIFs). Certain extensions also allow you to write NIFs in Rust. NIFs must be compiled, and their artifacts are typically not portable. This introduces complexity if you want to ship the same artifact to many different targets. You should be careful including NIFs in your project for this reason.
I would go farther and say that you should assume NIFs will be part of your project at some point, and design your build pipeline accordingly. Avoiding NIFs is not really necessary, though one should be careful of how many and how often you use them simply because of the risk of scheduler blockage, but that’s not really a deployment problem.
I generally recommend using Docker or Vagrant for local builds (i.e. set up a an environment to run mix release and produce the release tarball which you can then copy out of the build environment to wherever), and for teams, setting up a dedicated build server running something like Jenkins (or whatever, it could be Travis, CircleCi, just a dedicated build environment). The result of a build should be an artifact (in our case that is the release tarball) which can be copied to some kind of storage (whether that is S3, a file server, or a Docker image being pushed to a registry) or copied directly to the production server (I think holding on to each artifact for easy rollbacks/troubleshooting is worth storing them though). Actually deploying the release then becomes a matter of just copying the tarball to the right place, extracting it, and restarting the release.
Each of those steps can be completely automated, as well as performed by hand, which makes it easy to troubleshoot each step. If you are just getting into setting up a CI/CD pipeline, I’d start by getting the basic pieces in place, performing each step by hand, and then automate each piece until you are mostly just hitting a button to deploy to prod.
Configuration management is a big part of this, but that is heavily dependent on your setup, at the most basic level, environment variables work well, but you can automate things using any kind of config management tool or provisioning step (e.g. ansible/chef/puppet) or by hitting an external config service (e.g. etcd/consul/vault). Distillery 1.x doesn’t make things very easy in this regard, but Distillery 2.x will make it very easy to integrate different configuration providers. That should be coming out in the next couple weeks (by end of the month or first week of April I hope)
Things are different if you are building a product, or are working in an environment where you are at the mercy of another team who doesn’t care about helping you, but for the most part the above is all you really need to keep in mind to figure out an approach that works for you.
Wow! Remarkable feedback. It conveys establishing a “staged” manual pipeline before each part of the process: that appeals to me because it’s easy to describe to developers coming on board. I’ve now booked a build VM and am in the process of booking a staging VM.
I’m looking forward to Distillery 2’s arrival, and hoping strenuously it doesn’t do a typical development release (as in very late).
Yes, we’re building both a product and a service. Yes, we’ll soon grow to the stage of competing prima donna development teams. Please pray for us, we need all possible help. Or, for DevOps gurus with a strong Elixir background, we could pay you for CI/CD services!
For what it’s worth, Jenkins is old but it’s not a CI server it’s a job server. Jenkins is built to run a job, track it’s output, identify if it was successful or failed and then potentially do something afterwards. It also has a lot of plugins to deal with the output of jobs.
Everything else is just a matter of what triggers the job. Got commit, web request, another job, a timer, etc.
Just happens that all that a CI server does is run a build job and then do something with the output, which makes it a good fit for what Jenkins already does. I used it to consolidate and track cron jobs that were previously on about 20 different servers, log each output, alert on failure, etc. The scheduling utility even tracks the execution time and can automatically space the jobs so you are running a dozen things at 3am.
I used to dislike Jenkins as well, for pretty much the same reason (old and dated), until they recently released Blue Ocean (about 1-2 years ago). It’s no longer in the dark ages, and does CI/CD rather brilliantly. It’s free also. Check it out: https://jenkins.io/projects/blueocean/
If you still don’t like it, Rundeck is a free alternative job runner that has a modern UI (relatively) as well. It’s not designed explicitly for ci/cd, nor look as slick as blue ocean, but it’s not as janky as old jenkins and runs very dependably. It’s easy to write tasks with form inputs and bash syntax directly in the interface. It’s my current weapon of choice as a build server (but that’s primarily because I have the setup automated - I would go with BlueOcean if I was starting from scratch)
Great question and I found everyone’s answers very helpful as I’m a sys-admin new to the Erlang koolaid.
