Improvements to Elixir Deployments

kingdomcoder

Yes, Elixir is awesome.

But no, the Elixir/Phoenix deployment process is certainly not awesome and I think it should be a cause for concern.

It might well be the lowest DevEx point in our entire process (although, you could class this as OpEx )

I’d like to know what general direction we’re heading with deployment as a community.

However, here are a few specific questions:

Is it at all possible to achieve cross-platform mix releases in Elixir, akin to what’s possible with cross-platform compilation in Rust or Go? Will we always need to depend on Docker for this?
Rustler precompiled has made distributing ready-to-use, cross-platform Rust NIFs a breeze. What gaps need to be filled to achieve something similar with Elixir releases?
What will it take to achieve the promise of hot-code reloading – preserving in-memory state across deployments – in mix releases?
I’m aware that hot-code reloading has some technical quirks. So, what options do we have that are practical and robust enough to be first-class components of the mix release process?

What I wish exists is some “standard” deployment process that looks like this:

A GitHub release is created
Mix release files are generated for all (necessary) platforms and attached to the GitHub release
Running production node(s) fetch (as necessary) the mix releases and run them
In-memory state is preserved when the new mix release is applied

So, is work going on anywhere in the community that I’m unaware of that can make any of these components closer to reality?

I acknowledge the contributions of Fly.io, Gigalixir and Heroku to our current deployment story. The Phoenix documentation also does and tries to provide guides for deployment on each of these providers.

However, I think we will benefit greatly from a robust, vendor-agnostic deployment story with mix releases and I wonder what’s happening with this.

5 comments

#deployment #mix-release #mix-releases #rustler-precompiled

3 1743 5

2023-02-07 14:46:14 UTC

Showing Posts 1 to 5

Show Best Posts
Show All (oldest first)
Show All (newest first)

josevalim

Creator of Elixir

Hi @kingdomcoder!

I am trying to understand what are the benefits you are going after. I understand there are benefits of not having to rely on Docker and toolchains like Zig can bring us closer to cross-compilation but at the same time you seem to be asking for features that are not generally useful:

I can’t think of why server companies would need to cross-compile to multiple platforms and, if I am deploying to prod, I would rather build and test on machines as close as possible to production
I still think chasing hot-code upgrades is of little benefit to 99.9% of the companies and definitely should not be part of the “standard” deployment process (at least it should be opt-in)

There are benefits to cross-compilation, mostly for distributable Elixir apps such as CLI and for desktop, and projects like Burrito and Elixir Desktop are trying to provide answers here.

The point here is: deployment is a wide area and I think we should be specific which problems we are trying to solve and for which domain, especially because the solutions will change (and to avoid scope creep!)

However, I think we will benefit greatly from a robust, vendor-agnostic deployment story with mix releases and I wonder what’s happening with this.

Unfortunately, as much as I resisted to the idea, the answer to this is Docker, because outside of it there is no unified pipeline. We could tell people: hey, this is a script to build a release and this is a script to download and run it, but it doesn’t change the fact that getting those scripts to run on each different CI and each different provider is going to be different… unless you use Docker.

My understanding is that those deploying Go/Rust apps to servers are on a similar boat: Docker is coordinating the building and the running.

Post #1

LostKobrakai

As for 1.: Yes this is indeed possible and it‘s what nerves had been doing for years. They do rely on docker on non-linux systems, but afaik their CC toolchain does work without docker on linux. But this is a lot of work and nothing, which necessarily works out of the box for any dependency.

Post #2

schneebyte

There was a discussion in the build and packaging wg a while ago.
https://github.com/erlef/build-and-packaging-wg/issues/33#issuecomment-845077020

Also an interesting blog post with a proof of concept (with docker/containers):

Post #3

D4no0

Totally agree, the only thing that I don’t like about docker containers for elixir, is the fact that while you can select elixir version, OTP version is hardcoded, I always have to check the actual script to see the OTP version. It would be nice to be able to define the versions and docker to load the correct ones automatically.

Post #4