Yet another Elixir and Erlang docker image

ericmj · January 31, 2020, 12:43pm

The Hex team has been working on building our own Elixir and Erlang Docker images. We think they can be useful for the general community so I am sharing them in this post. The images are built by Bob which is the build service that provides the builds when you use Travis CI, GitHub CI, asdf, and other tools and services.

The reason we are building our own Docker images is because we have identified and run into a few issues with the existing image offerings that we are hoping to address with these images.

Issues with existing offerings

To sum it up:

Image tags are not immutable
Delay in availability of new versions
Cannot pick the combination of versions you want

Our main issue is that image tags are not immutable. If you pull the image elixir:1.9.4 you do not know which Erlang version or OS version you are getting. This is specially a problem when the tag the image points to changes to update the Erlang version. Recently Erlang introduced a breaking change in a minor version that made it an error when you started a client side SSL connection with the honor_cipher_order option, the Elixir image was updated to the new Erlang version which caused hackney to break and our deploys stopped working without any changes to either our dependencies or our Dockerfile.

The images we build with Bob are immutable and the versions used are explicit. An example of an Elixir image tag is 1.10.0-erlang-22.2.4-alpine-3.11.3, here we can see that the Elixir, Erlang and Alpine versions are explicit and there is no reason to update any versions behind the scenes.

Sometimes there are delays in the availability of images for new releases. As I am writing this the new images for Elixir 1.10.0 are not yet available on the official Docker images even though 1.10.0 was released 3 days ago. The images from Bob are built automatically and should be available within 30 minutes of tagged releases.

Our final issue has been that we cannot pick the exact versions we want. From the official images I can install Elixir 1.9.4 but I have no idea what Erlang or OS version will be used. Sometimes we want to use the latest features from Erlang or stay on an older version for stability but this is not an option. The images from Bob are built with all compatible versions of the underlying OS, Erlang, and Elixir.

Possible downsides with Bob’s images

First off, they are currently experimental. This means you should take care before running them in production. We are running them in production for all Hex services without issue so far. While they are experimental the immutability guarantee of image tags may not hold if we find issues that require such changes.

We currently only provide images on the latest Alpine versions. In the future we may build on more systems.

Since we build for all compatible OS, Erlang, and Elixir versions we produce many, many images that use up disk spaces. We haven’t run into any limitations on DockerHub yet, but it is possible we do in the future. As far as I can tell there are no documented limitations.

To build the images faster we use the pre-compiled versions of Erlang and Elixir that Bob already provides. Because of this the original Makefiles are not available so we cannot use make install to install files in the correct locations. Currently we install Elixir and Erlang in the root of the filesystem at /elixir and /erlang respectively. This is obviously not ideal so if anyone has a solution to this problem we are all ears.

Links

To read more about Bob’s docker images go to our README: https://github.com/hexpm/bob#docker-images. Links to the DockerHub repositories can be found here: https://hub.docker.com/r/hexpm/erlang and here: https://hub.docker.com/r/hexpm/elixir.

Phillipp · January 31, 2020, 12:50pm

Nice! How about versions for ARM, so we can easily use it on a Raspberry Pi?

ericmj · January 31, 2020, 1:18pm

We need to make sure the current images work correctly first but I think we can support ARM in the feature. We don’t have ARM builds of OTP so the first step would be to add cross-compilation to ARM in the Bob OTP builds.

Phillipp · January 31, 2020, 1:24pm

Cool. I am looking forward to it. It’s always a real struggle if you want to use Docker on a Raspberry Pi because many things are just not available for ARM. I think having this sorted out for Elixir will be very helpful.

entone · January 31, 2020, 3:09pm

The “not immutable” image tags have bitten me/us several times. Excited for officially supported community versions. Great work!

mhs · January 31, 2020, 4:01pm

Any plans for Debian (stable) slim builds? [buster, at time of this post]

I like Alpine but I’ve actually switched back to buster-slim over subtle concerns around glibc / musl and DNS.

ericmj · January 31, 2020, 4:48pm

We want to support more OSes and CPU architectures when we know that the images are stable. The first step would be to build OTP against those architectures and OSes. Here you can find our existing linux builds [1], any contributions to support more are welcome.

[1] bob/priv/scripts/otp at main · hexpm/bob · GitHub

foggy · January 31, 2020, 6:54pm

This is great news. We’ve been internally pinning the official Docker images ever since the incident with the ssl dep and erlang broke all of our stuff (twice). This will likely save us that trouble in the long run.

sorentwo · January 31, 2020, 11:09pm

Having been stung by the mutable erlang changes (hackney ssl), mutable alpine version (libvips mismatch), and an annoying lag between version release and docker availability; I’m all for this.

I trust the hex team to provide timely and consistent builds more than I trust the unknown (to me) authors of the official image.

Thanks!

Exadra37 · February 1, 2020, 12:14am

Oh it seems that I was not the only one in need of immutable images

╭─exadra37@thinkpap-l460 ~/Developer/Projects/elixir-docker-stack  ‹dev-new-approach› 
╰─➤  sudo docker image ls | grep -i erlang
exadra37/phoenix-dev       1.4.11_elixir-1.9.4_erlang-22.2.1_git   23d45190fd25        3 weeks ago         814MB
exadra37/elixir-dev        1.9.4_erlang-22.2.1_git                 98f3ef6b30ac        3 weeks ago         713MB
exadra37/erlang-dev        22.2.1_git                              96c72e47095b        3 weeks ago         707MB
exadra37/phoenix-dev       1.4.10_elixir-1.9.4_erlang-22.1.6_git   46d86f30f487        3 weeks ago         813MB
exadra37/elixir-dev        1.9.4_erlang-22.1.6_git                 14df71ee5f57        3 weeks ago         712MB
exadra37/erlang-dev        22.1.6_git                              401ebb0c0162        3 weeks ago         706MB

But in my case I have built them as part of an Elixir Docker Stack that automates a lot of stuff for my developer needs, and I compile Erlang, Elixir and Phoenix directly from Github, and I am based on Debian, mainly because of Observer not working in some Alpine based images.

Thanks for this work. I need to try it out, and compare it with my approach, because Alpine based images have not worked for me.

Exadra37 · February 4, 2020, 3:24pm

It’s not about Docker, but about how you write the Dockerfile.

If you want a truly immutable docker image until the OS level, then you need to pin everything you install with apt or whatsoever package manager you use to install things in the OS.

But if you want to extend another docker image and have it pinned, you cannot use the tag, instead you need to use the digest hash of it. Read more in the official docs:

Pull an image by digest (immutable identifier)

So far, you’ve pulled images by their name (and “tag”). Using names and tags is a convenient way to work with images. When using tags, you can docker pull an image again to make sure you have the most up-to-date version of that image. For example, docker pull ubuntu:14.04 pulls the latest version of the Ubuntu 14.04 image.

In some cases you don’t want images to be updated to newer versions, but prefer to use a fixed version of an image. Docker enables you to pull an image by its digest . When pulling an image by digest, you specify exactly which version of an image to pull. Doing so, allows you to “pin” an image to that version, and guarantee that the image you’re using is always the same.

Dockerfile Example from docs:

FROM ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2

So no matter how many years we go into the future, the image will be always exactly the same you download since it was built. It’s like a Git commit, if it changes the hash is not the same anymore.

mmmries · February 16, 2020, 5:06am

Thanks @ericmj, and hex team for building these. I just used one and it was so nice to not have to check what Alpine and ERTS version were being used. On top of that, the images are really small

jayjun · March 5, 2020, 4:10am

Official Node, Python and Ruby images generally only have buster, buster-slim, stretch and stretch-slim variants in addition to Alpine ones.

I prefer these Debian variants over Ubuntu.

I’m guessing Bob builds against Ubuntu mainly for clients, but I’d venture to say more run Debian in containers.

ericmj · March 10, 2020, 5:27pm

We have added images based on Ubuntu LTS versions. They are based on the following image tags:

ubuntu:bionic-20200219
ubuntu:xenial-20200212
ubuntu:trusty-20191217

Example tags:

hexpm/erlang:22.2.8-ubuntu-bionic-20200219
hexpm/elixir:1.10.2-erlang-22.2.8-ubuntu-xenial-20200212

We also solved the issue of installing into the root of the filesystem in /erlang and /elixir by not using the precompiled builds and instead use make install to install in the correct locations according to the Filesystem Hierarchy Standard.

Next we are planning to add debian images. We also want to provide images for ARM architectures, but since we are no longer using the precompiled builds the option of cross-compiling is not possible. This means we either have to emulate ARM or use ARM hardware. I have tested the emulation option using docker buildx, but it seems far too slow, and we currently do not have access to ARM machines.

praveenperera · March 10, 2020, 6:23pm

Thanks a lot for this! I love this approach.

It also makes it very easy for me to know which version of Alpine to pull from on my release container.

ericmj · March 31, 2020, 8:04pm

We have added builds based on debian and based on the ARM v8 architecture (called arm64 on Docker). The new debian images are based on the following image tags:

debian:buster-20200224
debian:stretch-20200224
debian:jessie-20200224

We use “fat manifests” to be able to serve both amd64 and arm64 using the same image tag names, that means you can docker pull hexpm/erlang:22.3-alpine-3.11.3 and docker will figure out the correct architecture to pull for your system. We provide arm64 images for all the same versions we provide amd64 images as long as we are able to build Erlang and Elixir on the given versions, but some older distributions and Erlang versions do not fully support ARM.