Checkout Nanobox.io. You can deploy to DO. You could probably learn a lot by looking at their open source stuff.
I currently use docker with release builds. I’d love, love, love to do hot upgrades, but I don’t.
What I don’t really understand is why docker is so incompatible with elixir and its release style. Inside a docker container, you have a single vm running a single application on it and nothing else. It should be incredibly simple to build an upgrade from that and then apply that upgrade to the running container. It is a highly specific use case and it takes away a ton of assumptions and edge cases that normally would have to be taken into account.
I could imagine an image on hub.docker designed for this, but it is a little beyond my knowledge.
@mindreader You can hot upgrade a running container but you just have to do it from ‘within’ that running container. In addition you want to make sure you have an upgraded image for whenever it restarts.
The way described in this video seems to be more aligned with a container environment. Rather than relying on being able upgrade the application around the existing state, state is migrated from the old node to the new one. One advantage is that the deployment unit can be resurrected on an entirely different server.
The approach requires design for migrate-ability but even hot upgrades require a certain amount of forethought.
2 Likes
They have been silent for months, no one knows what’s going on at Nanobox.io. There is another thread about them here, I used to be a fan but no longer.
That’s scary, I’m still using them.
I’m using docker and docker-compose with nginx as a load balancer. Within docker-compose I have a definition for two containers for the app (app_a and app_b) and one for postgres.
On the production host, I have a tiny shell-script which:
- pulls the new images and restarts the container for
app_a - when app_a is up and running, it does the same for app_b
Although I have to solve some race conditions yet, this works fine unless the upgrade needs to migrate the database in a way where the old version can’t continue working with the migrated DB-schema. In this case, I have to stop both containers before upgrading.
To be honest, I do not expect that this approach will work in heavy production conditions but at the moment our customer doesn’t work nighttimes, thus I can live with a one-minute downtime if necessary.
I appreciate the methods mentioned in this thread. I will try Distillery once I need to upgrade without any downtime. Anyhow I will stay with Docker because I feel so comfortable testing on my machine (OSX) and can be (99.99…%) sure it will work on the production machine (Linux) as well. I even sent images to the customer for previewing on their Windows machines.
Kubernetes seems to be too much effort for a tiny app like mine.
1 Like
I go back and forth myself. Current setup is droplets + edeliver when at single server stage and k8s once I need more than one app server + lb.
We are deploying all our infrastructure with Docker (postgres, elixir backend, frontend SPA apps, some other services). It’s never being a pain for me to manage all the stack myself without devops team.
For a small / mid-sized project with 1-3 nodes (without scaling) I would recommend to give a try to Docker Swarm (We was managing with Kubernetes from start but moved to Swarm because vast default resource consumption/claim by Kubernetes and unnecessary over-complication for our stack requirements).
The whole process is pretty straightforward
- commit
- build docker image and push to registry
- notify swarm about new image
The image building process depends on your team, it’s either some proper build with Distillery or just mix phx.server to run the application.
Also, we don’t deploy manually stage / production servers, just git push to trigger GitLab CI pipelines.
One separate issue with Docker deployments is image building time if you are using some cloud CI because inability to use cache from previous builds and docker-in-docker (dind). We solved it by hosting our own worker for GitLab builds.
This issue doesn’t relate to docker deployments. Best practice for such situations is to write migration which doesn’t break old code. You can do it many ways where one of is splitting migration to several steps.
Good point. Which leads me to the question if and how a Docker-approach can help with such a multi-step-upgrade.
Docker has nothing to do with multi-step-upgrade, it’s just a tool/platform to unify process of deploying and running software.
It can help you to unify managing process for all software (backend, frontend, storage) and isolate environment for a running program. Smooth deployments with proper failure rollbacks addresses development process (and code logic) rather than hosting platform runing the program.
A docker image is built in layers. At the point that that image is complete, all of those layers are write protected - you can’t change them. If you build your Elixir system into the image then its code cannot be changed later. You could say it’s baked into the image.
Later, when you create a container based on the image, the runtime adds a single writeable layer on top of the last layer in the image. When you do things that change the file system, those changes are made in this writable layer.
You can do hot upgrades with Elixir if you build an image, then run a container, then install the release into the running container. When you do that the Elixir/OTP application (and probably the ERTS) are written into the writable layer of the container and are fungible. But setting up your release pipeline to account for the extra step of firing up a container then installing the application into that container is added complexity. If you are running your application as a cluster of nodes - you would need to handle the install step whenever a node goes down and your orchestration system has to fire it back up. Couple that with the inherit complexity of designing your code to handle hot upgrades and it all adds up,
2 Likes
I’ve been deploying to Digital Ocean with this setup below:
https://www.shanesveller.com/blog/2018/11/13/kubernetes-native-phoenix-apps-part-2/
I probably will never use Kubernetes(I’m building an elixir backend on my volunteer time not for pay), but the Docker deployment in Part 2 above works for me(so far).
EDIT: I forgot to mention that I’m using Traefik as a proxy/let’s encrypt manager. Works great so far with the umbrella app.
Michael
Even though it seems to be talked down on quite a bit I am still very happy with hot-updating. So long as your not updating process state it is honestly really easy using distillery.
My current process is as follows and works really well for me at https://appdoctor.io
- builds trigger on git master branch
- builds runs on codeship
- runs tests
- pulls last artifact from previous build(so that distillery can build out a hot-upgrade)
- run distillary release --upgrade
- send upgrade artifact to digitalocean spaces
Later I have a deploy script in elixir where I do something like Ssh.deploy(“us_east”,“version”) this script will determine if its going to be a hot update or fresh install and run the commands accordingly.
I have a lot of timing logic working globally in state(ie run this users next api test in 50 minutes). Because I can use hot updates its a really simple path to not mess with that state or connected web-sockets. It kind of…just works.
6 Likes
