Yet another Elixir and Erlang docker image

The Hex team has been working on building our own Elixir and Erlang Docker images. We think they can be useful for the general community so I am sharing them in this post. The images are built by Bob which is the build service that provides the builds when you use Travis CI, GitHub CI, asdf, and other tools and services.

The reason we are building our own Docker images is because we have identified and run into a few issues with the existing image offerings that we are hoping to address with these images.

Issues with existing offerings

To sum it up:

  1. Image tags are not immutable
  2. Delay in availability of new versions
  3. Cannot pick the combination of versions you want

Our main issue is that image tags are not immutable. If you pull the image elixir:1.9.4 you do not know which Erlang version or OS version you are getting. This is specially a problem when the tag the image points to changes to update the Erlang version. Recently Erlang introduced a breaking change in a minor version that made it an error when you started a client side SSL connection with the honor_cipher_order option, the Elixir image was updated to the new Erlang version which caused hackney to break and our deploys stopped working without any changes to either our dependencies or our Dockerfile.

The images we build with Bob are immutable and the versions used are explicit. An example of an Elixir image tag is 1.10.0-erlang-22.2.4-alpine-3.11.3, here we can see that the Elixir, Erlang and Alpine versions are explicit and there is no reason to update any versions behind the scenes.

Sometimes there are delays in the availability of images for new releases. As I am writing this the new images for Elixir 1.10.0 are not yet available on the official Docker images even though 1.10.0 was released 3 days ago. The images from Bob are built automatically and should be available within 30 minutes of tagged releases.

Our final issue has been that we cannot pick the exact versions we want. From the official images I can install Elixir 1.9.4 but I have no idea what Erlang or OS version will be used. Sometimes we want to use the latest features from Erlang or stay on an older version for stability but this is not an option. The images from Bob are built with all compatible versions of the underlying OS, Erlang, and Elixir.

Possible downsides with Bob’s images

First off, they are currently experimental. This means you should take care before running them in production. We are running them in production for all Hex services without issue so far. While they are experimental the immutability guarantee of image tags may not hold if we find issues that require such changes.

We currently only provide images on the latest Alpine versions. In the future we may build on more systems.

Since we build for all compatible OS, Erlang, and Elixir versions we produce many, many images that use up disk spaces. We haven’t run into any limitations on DockerHub yet, but it is possible we do in the future. As far as I can tell there are no documented limitations.

To build the images faster we use the pre-compiled versions of Erlang and Elixir that Bob already provides. Because of this the original Makefiles are not available so we cannot use make install to install files in the correct locations. Currently we install Elixir and Erlang in the root of the filesystem at /elixir and /erlang respectively. This is obviously not ideal so if anyone has a solution to this problem we are all ears.

Links

To read more about Bob’s docker images go to our README: https://github.com/hexpm/bob#docker-images. Links to the DockerHub repositories can be found here: https://hub.docker.com/r/hexpm/erlang and here: https://hub.docker.com/r/hexpm/elixir.

39 Likes

Nice! How about versions for ARM, so we can easily use it on a Raspberry Pi?

How do other programming language runtimes fix this while still using Docker’s official images?

For example Python, Ruby and Node all have specific OS version tags. Such as XX-slim-buster or XX-slim-stretch. They also usually get new releases built quite quickly.

I opened an issue on GitHub 25 days ago about getting more locked down tag versions but no one replied. Is this just a matter of the person in charge of the official Elixir Docker image going MIA?

I don’t know how the pipeline works exactly but I’m pretty sure it comes down to someone related to the technology for each official image being in charge of managing builds for that image. Docker just gives them write access to the git repo. In other words, it’s a community member of the programming run-time handling this, not a Docker employee.

1 Like

We need to make sure the current images work correctly first but I think we can support ARM in the feature. We don’t have ARM builds of OTP so the first step would be to add cross-compilation to ARM in the Bob OTP builds.

1 Like

Cool. I am looking forward to it. It’s always a real struggle if you want to use Docker on a Raspberry Pi because many things are just not available for ARM. I think having this sorted out for Elixir will be very helpful.

Even if you have XX-slim-buster you don’t know which version of buster or what their update policy is. Operating systems are usually more careful about making breaking changes than language runtimes are so it may be less of an issue to update to the latest OS version.

The way official Docker images are built is different to how we do it. Official Docker images are built on Docker’s own build servers so the maintainer of the image has to tell Docker exactly which dockerfiles and tags to build (this makes sense for Docker since you don’t want someone unknown to push images). But it would not be feasible for us since we build so many different version combinations, instead Bob looks up all Erlang and Elixir versions and then use a compatibility table to build the correct combinations automatically when new versions are available.

This is how I understand it as well. The person maintaining the official docker image may not officially be part of Docker or officially part of the team for the language.

1 Like

Do you build more combos than the Python image? That one has so many.

I think ditching the official Elixir image should be a last resort move since official images carry a lot of weight to them and Docker also has a bias for them in search results. Every official image will be listed before others. Also from a social POV, it’s just what people expect for the “community blessed” version.

This might cause a lot of confusion for new people using Docker with Elixir because they will see an outdated official Elixir image. Also it means everyone using Elixir with Docker currently will need to change all of their client code to reference the new location.

If you could work around the version combination issue would that solve the main problem, since it sounds like turn around time is controlled by the maintainer of the image (ie. if 1.10 drops, if the maintainer is on the ball it could be up on the Docker Hub in a matter of hours).

So far I haven’t heard of any organization not being able to use the official images due to limitations of the platform so I’m wondering if there’s a way to still use it.

If you have any questions you want me to forward to Docker about what would need to be done to make this work with official images let me know. I’m not an employee, but I am a “Docker captain”. It basically means I can talk to Docker employees and engineers and ask them anything. I’m sure they would be interested in also knowing why some people are thinking about not using official images.

If you can list out the exact problems, limitations and concerns that would be great.

1 Like

Yes, they seem to have around ~1000 tags and we currently have ~4000 tags.

I should probably have clarified this in the original post. We are not looking to replace any existing images, we provide an alternative for people that have run into the same issues we have with the existing images. If you are more comfortable using the official images and you haven’t had any issues with them you should continue using them.

It seems that the maintainer needs to update the repository the images are based on [1]. Then the new tags need to be accepted and built by Docker also [2].

I don’t know exactly how the official images work so it’s hard to say what the limitations are. I listed the problems with the existing Elixir and Erlang images in the original post, is there anything you would like me to expand on?

[1] https://github.com/c0b/docker-elixir/commit/e2d20313fefcd13b8d3b1bcd94df60bd4f02a0b3
[2] https://github.com/docker-library/official-images/pull/7375

Based on those reference links it looks like it took around 15 hours for the PR to exist, get reviewed by Docker and now it’s on the Docker Hub. The multi-day wait is because it looks like the PR was stalled by the non-Docker community member in charge of the official Elixir image for a few days.

I think that will cause a bit of confusion too. Shouldn’t we try to focus everything into having 1 solution that is optimal? Otherwise people will be conflicted with choice, and honestly unless they dig around issues and forum posts they won’t even know your alternative exists since the default standard is to roll with the official image.

It sounds like almost all of those problems could be addressed by having a different official Elixir Docker image maintainer unless I’m missing something? Could any tooling be written to create those super explicit tags using the Docker Hub’s builder?

More options is not not necessarily a bad thing. These images existing should not stop the community from using and improving the official images. We built these images because we did not see a way to solve the problems using Docker’s official build process.

This is not something I have looked into in much detail. I am afraid it could be a bit spammy for the Docker team since all tags need to be manually accepted by them. But if we could solve that and build official images then that would be the ideal solution of course.

The “not immutable” image tags have bitten me/us several times. Excited for officially supported community versions. Great work!

2 Likes

I’m not sure how it works either but the Node image has 2,500+ tags going by the list of tags on the Docker Hub (25 per page and 103 pages). I’m sure they have systems in place to automate dealing with that since they have hundreds of official images using thousands of tags each.

I’m still trying to figure out the exact mass tag problem here so I can pass the info along to the Docker folks in charge of accepting new tags. Like you, I would much prefer immutable tags too, but it sounds like that could be done with the official images. It’s just the process of assembling the Dockerfiles for each one that’s a pain? But this could be solved with a tool without Docker having to do anything on their side of things?

Any plans for Debian (stable) slim builds? [buster, at time of this post]

I like Alpine but I’ve actually switched back to buster-slim over subtle concerns around glibc / musl and DNS.

1 Like

The underlying problem is that Docker images are not composable. I cannot pick the OS version, Erlang version, and Elixir version without someone already having built that exact combination of versions. Because of this we have to prebuild many different combinations of versions producing many different tags.

Right now we build against two Alpine versions and we already have over 4000 tags, but if we are to build against Ubuntu, Debian, Slim, etc., with multiple different versions it’s just not going to scale if there is a manual step by humans involved. If I could push my own built official images to DockerHub or if there was fully automatic build process we could use that would be great. I don’t really know how official images work so I cannot say if this feasible or what the problems would be. Sorry, that I cannot help more.

1 Like

We want to support more OSes and CPU architectures when we know that the images are stable. The first step would be to build OTP against those architectures and OSes. Here you can find our existing linux builds [1], any contributions to support more are welcome.

[1] https://github.com/hexpm/bob/tree/master/priv/scripts/otp

I don’t think you’d be able to push them direct but I could ask about an automatic build process.

I didn’t realize that was without Debian too. Personally I’ve been running Debian “slim” for my base images for a long time now (after having used Alpine in the past). I hope to see Debian “slim” support in your builds one day.

This is great news. We’ve been internally pinning the official Docker images ever since the incident with the ssl dep and erlang broke all of our stuff (twice). This will likely save us that trouble in the long run.

Having been stung by the mutable erlang changes (hackney ssl), mutable alpine version (libvips mismatch), and an annoying lag between version release and docker availability; I’m all for this.

I trust the hex team to provide timely and consistent builds more than I trust the unknown (to me) authors of the official image.

Thanks!

Oh it seems that I was not the only one in need of immutable images :slight_smile:

╭─exadra37@thinkpap-l460 ~/Developer/Projects/elixir-docker-stack  ‹dev-new-approach› 
╰─➤  sudo docker image ls | grep -i erlang
exadra37/phoenix-dev       1.4.11_elixir-1.9.4_erlang-22.2.1_git   23d45190fd25        3 weeks ago         814MB
exadra37/elixir-dev        1.9.4_erlang-22.2.1_git                 98f3ef6b30ac        3 weeks ago         713MB
exadra37/erlang-dev        22.2.1_git                              96c72e47095b        3 weeks ago         707MB
exadra37/phoenix-dev       1.4.10_elixir-1.9.4_erlang-22.1.6_git   46d86f30f487        3 weeks ago         813MB
exadra37/elixir-dev        1.9.4_erlang-22.1.6_git                 14df71ee5f57        3 weeks ago         712MB
exadra37/erlang-dev        22.1.6_git                              401ebb0c0162        3 weeks ago         706MB

But in my case I have built them as part of an Elixir Docker Stack that automates a lot of stuff for my developer needs, and I compile Erlang, Elixir and Phoenix directly from Github, and I am based on Debian, mainly because of Observer not working in some Alpine based images.

Thanks for this work. I need to try it out, and compare it with my approach, because Alpine based images have not worked for me.

I think they just don’t care. Also caching comes into play.

When I was building my application Dockerfiles the fact that the versions are not pinned gave me some anxiety since I knew about the bugged minor Erlang release from the release notes (not sure if it’s the same issue as mentioned by Eric). I’m happy that this is getting addressed, a bit sad that Docker doesn’t support this. The best you can do is to pin Elixir X.Y.Z to Erlang A.B.C and that to Alpine M.N.O, but that’s not really flexible.