Arm64 Dockerfile failing: ERROR [builder 6/17] RUN mix deps.get --only prod

sergio-ocon · July 29, 2023, 5:01pm

Hi,

I am trying to build a multiple architecture image using Docker build.

> docker build -t chargio/liveview_counter --push --platform linux/arm64  .

I get an error (139) running this
> [+] Building 7.3s (16/28)                                                                                                                                                                                                                                                                                                                              docker:desktop-linux
>  => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                      0.0s
>  => => transferring context: 1.35kB                                                                                                                                                                                                                                                                                                                                    0.0s
>  => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                   0.0s
>  => => transferring dockerfile: 2.54kB                                                                                                                                                                                                                                                                                                                                 0.0s
>  => [internal] load metadata for docker.io/library/debian:bookworm-20230612-slim                                                                                                                                                                                                                                                                                       0.9s
>  => [internal] load metadata for docker.io/hexpm/elixir:1.15.3-erlang-26.0.2-debian-bookworm-20230612-slim                                                                                                                                                                                                                                                             1.0s
>  => [builder  1/17] FROM docker.io/hexpm/elixir:1.15.3-erlang-26.0.2-debian-bookworm-20230612-slim@sha256:9fdcffc63a03bf77bf437510634ee7ad182135fc246283986c58fd8686c89326                                                                                                                                                                                             0.0s
>  => => resolve docker.io/hexpm/elixir:1.15.3-erlang-26.0.2-debian-bookworm-20230612-slim@sha256:9fdcffc63a03bf77bf437510634ee7ad182135fc246283986c58fd8686c89326                                                                                                                                                                                                       0.0s
>  => [internal] load build context                                                                                                                                                                                                                                                                                                                                      0.1s
>  => => transferring context: 73.36kB                                                                                                                                                                                                                                                                                                                                   0.1s
>  => [deploy 1/6] FROM docker.io/library/debian:bookworm-20230612-slim@sha256:d8f9d38c21495b04d1cca99805fbb383856e19794265684019bf193c3b7d67f9                                                                                                                                                                                                                          0.0s
>  => => resolve docker.io/library/debian:bookworm-20230612-slim@sha256:d8f9d38c21495b04d1cca99805fbb383856e19794265684019bf193c3b7d67f9                                                                                                                                                                                                                                 0.0s
>  => CACHED [deploy 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales   && apt-get clean && rm -f /var/lib/apt/lists/*_*                                                                                                                                                                                                          0.0s
>  => CACHED [deploy 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                                                                                                                                                                                                                                                               0.0s
>  => CACHED [deploy 4/6] WORKDIR /app                                                                                                                                                                                                                                                                                                                                   0.0s
>  => CACHED [deploy 5/6] RUN chown nobody /app                                                                                                                                                                                                                                                                                                                          0.0s
>  => CACHED [builder  2/17] RUN apt-get update -y && apt-get install -y build-essential git     && apt-get clean && rm -f /var/lib/apt/lists/*_*                                                                                                                                                                                                                        0.0s
>  => CACHED [builder  3/17] WORKDIR /app                                                                                                                                                                                                                                                                                                                                0.0s
>  => CACHED [builder  4/17] RUN mix local.hex --force &&     mix local.rebar --force                                                                                                                                                                                                                                                                                    0.0s
>  => CACHED [builder  5/17] COPY mix.exs mix.lock ./                                                                                                                                                                                                                                                                                                                    0.0s
>  => ERROR [builder  6/17] RUN mix deps.get --only prod                                                                                                                                                                                                                                                                                                                 6.1s
> ------
>  > [builder  6/17] RUN mix deps.get --only prod:
> 6.023 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> 6.036 Segmentation fault
> ------
> Dockerfile:39
> --------------------
>   37 |     # install mix dependencies
>   38 |     COPY mix.exs mix.lock ./
>   39 | >>> RUN mix deps.get --only $MIX_ENV
>   40 |     RUN mkdir config
>   41 |
> --------------------
> ERROR: failed to solve: process "/bin/sh -c mix deps.get --only $MIX_ENV" did not complete successfully: exit code: 139

Blockquote

The same works great when using amd64.

I am using the Dockerfile generated by mix release --docker. And I’ve changed the debian version several times to use ‘bullseye’ - the default, ‘buster’, and ‘bookworm’, with the same result.

Do you know if there is some problem with the arm64 version of the image?

dimitarvp · July 29, 2023, 5:28pm

Can you show us which exact container you’re using as a base?

sergio-ocon · July 29, 2023, 5:29pm

I get the same error in the build phase using different docker images (as found here: Docker)

Darwin iMac-de-Sergio.local 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64

I’ve basically tried the non-slim version and it gets the same error, and I’ve found some similar errors with people trying to do the opposite (running amd64 containers in arm64 computers). I am using qemu 8.0.3 installed using brew.

The dockerfile:

# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
# instead of Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
# This file is based on these images:
#
#   - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
#   - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20230612-slim - for the release image
#   - https://pkgs.org/ - resource for finding needed packages
#   - Ex: hexpm/elixir:1.15.3-erlang-26.0.2-debian-bullseye-20230612-slim
#
ARG ELIXIR_VERSION=1.15.4
ARG OTP_VERSION=26.0.2
ARG DEBIAN_VERSION=bookworm-20230612

ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

COPY lib lib

COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE} as deploy

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

WORKDIR "/app"
RUN chown nobody /app

# set runner ENV
ENV MIX_ENV="prod"

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/live_view_counter ./

USER nobody

CMD ["/app/bin/server"]

(the version has rotated between different versions of Debian + slim)

The application runs perfectly in x86_64. You can find the code here:

And the description of it here:
https://medium.com/@chargio/how-to-easily-run-your-elixir-application-in-a-local-kubernetes-using-docker-desktop-f0c1ccfd49e6

dimitarvp · July 29, 2023, 5:53pm

I scanned your linked blog post and noticed this:

 => [internal] load metadata for docker.io/hexpm/elixir:1.15.3-erlang-26.0.2-debian-bullseye-20230612-slim                                                                                                                                                                                                                                                             1.5s

When the image really should begin with hexpm/elixir-arm64 and not with hexpm/elixir.

Just a blind shot though, assuming you haven’t tried it.

sergio-ocon · July 29, 2023, 5:53pm

Tried changing from Docker Desktop to Rancher Desktop with the same result when using QEMU and the Apple Virtualization framework (VZ), experimental.

sergio-ocon · July 29, 2023, 6:47pm

The blog post is using amd64 as it is being deployed in a cluster in the cloud so it is ok.
Now I am trying to deploy the image in a raspberry pi (the cloud cluster is no longer active), and make the app multi-container, and I need to build the image for arm64 to make it run there. Unfortunately, it is not working. I am trying to understand if the problem is the image itself or qemu, because I no longer have an M1 machine to compile it for arm64.

[Corrected - it was amd64, not arm64]

dimitarvp · July 29, 2023, 6:53pm

Not sure I get this, if the FROM Dockerfile clause is not pointing at the right arch how can it be okay?

sergio-ocon · July 29, 2023, 7:58pm

My mistake, the blogpost was written first time on an M1, but compiled by a friend and updated on an x86_64 Linux machine, then the second one on an X86_64 Mac as I no longer have the M1. Now I am trying to build the ARM version on the X86_64 machine to run it on K3s on a Raspberry Pi 4B, and it is failing.

I will try to do it tomorrow inside the Raspberry, even if it takes forever, and see if it works as expected.

pdgonzalez872 · July 29, 2023, 8:05pm

hi @sergio-ocon can you please add these vars to your setup? Mix deps.get memory explosion when doing cross-platform Docker build - #2 by pdgonzalez872

dimitarvp · July 29, 2023, 8:05pm

I feel we’re talking past each other here.

Can you just try replacing this:

FROM hexpm/elixir:1.15.3-erlang-26.0.2-debian-bullseye-20230612-slim

with this:

FROM hexpm/elixir-arm64:1.15.3-erlang-26.0.2-debian-bullseye-20230612-slim

?

This should help you build it on any host, no?

cschmatzler · July 29, 2023, 9:46pm

Add ENV ERL_FLAGS="+JPperf true" to your Dockerfile.

sergio-ocon · July 30, 2023, 10:00am

This made the trick. Now it is running and my image is published.

My Dockerfile:

# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
# instead of Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
# This file is based on these images:
#
#   - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
#   - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20230612-slim - for the release image
#   - https://pkgs.org/ - resource for finding needed packages
#   - Ex: hexpm/elixir:1.15.3-erlang-26.0.2-debian-bullseye-20230612-slim
#
ARG ELIXIR_VERSION=1.15.4
ARG OTP_VERSION=26.0.2
ARG DEBIAN_VERSION=bookworm-20230612-slim

ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"
ENV ERL_FLAGS="+JPperf true"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

COPY lib lib

COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE} as deploy

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

WORKDIR "/app"
RUN chown nobody /app

# set runner ENV
ENV MIX_ENV="prod"

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/live_view_counter ./

USER nobody

CMD ["/app/bin/server"]

There is bug in Docker that prevents export of images to work if you are not using the base image too in the same command, so defining the platform as linux/arm64 on its own fails, you need to define both local (amd64) and the cross one (arm64)

% docker build -t chargio/liveview_counter --push  --platform linux/amd64,linux/arm64  .

sergio-ocon · November 15, 2024, 7:11am

One year later and the problem is still there. If I cross-compile using Github actions, the build fails when I don’t add JPperf true.

The logs shows this:

#19 16.54 Error while loading project :thousand_island at /app/deps/thousand_island
#19 16.57 {error,badarg,[{ets,lookup_element,[elixir_modules,'Elixir.Telemetry.Metrics',2],[{error_info,#{cause=>badkey,module=>erl_stdlib_errors}}]},{elixir_module,data_tables,1,[{file,"src/elixir_module.erl"},{line,27}]},{'Elixir.Kernel.Typespec',deftypespec,6,[{file,"lib/kernel/typespec.ex"},{line,136}]},{'Elixir.Inspect.PID',module_info,0,[]},{'Elixir.Exception',format_banner,3,[{file,"lib/exception.ex"},{line,148}]},{'Elixir.Kernel.CLI',format_error,3,[{file,"lib/kernel/cli.ex"},{line,107}]},{'Elixir.Kernel.CLI',print_error,3,[{file,"lib/kernel/cli.ex"},{line,177}]},{'Elixir.Kernel.CLI',exec_fun,2,[{file,"lib/kernel/cli.ex"},{line,165}]}]}
#19 16.57 Runtime terminating during boot ({badarg,[{ets,lookup_element,[elixir_modules,'Elixir.Telemetry.Metrics',2],[{error_info,#{cause=>badkey,module=>erl_stdlib_errors}}]},{elixir_module,data_tables,1,[{file,"src/elixir_module.erl"},{line,27}]},{'Elixir.Kernel.Typespec',deftypespec,6,[{file,"lib/kernel/typespec.ex"},{line,136}]},{'Elixir.Inspect.PID',module_info,0,[]},{'Elixir.Exception',format_banner,3,[{file,"lib/exception.ex"},{line,148}]},{'Elixir.Kernel.CLI',format_error,3,[{file,"lib/kernel/cli.ex"},{line,107}]},{'Elixir.Kernel.CLI',print_error,3,[{file,"lib/kernel/cli.ex"},{line,177}]},{'Elixir.Kernel.CLI',exec_fun,2,[{file,"lib/kernel/cli.ex"},{line,165}]}]})
#19 16.59
#19 16.59 Crash dump is being written to: erl_crash.dump...done
#19 ERROR: process "/bin/sh -c mix deps.compile" did not complete successfully: exit code: 1

My Dockerfile (updated and tested so it works):

# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
# instead of Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
# This file is based on these images:
#
#   - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
#   - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20241016-slim - for the release image
#   - https://pkgs.org/ - resource for finding needed packages
#   - Ex: hexpm/elixir:1.17.3-erlang-27.1.2-debian-bullseye-20241016-slim
#
ARG ELIXIR_VERSION=1.17.3
ARG OTP_VERSION=27.1.2
ARG DEBIAN_VERSION=bullseye-20241016-slim

ARG BUILDER_IMAGE="docker.io/hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${BUILDER_IMAGE} AS builder

ENV SRC_CODE="."

# Setting up locale so it does not complain about misconfigured latin1
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
ENV CODE=/app

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR ${CODE}

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"
# Add JPperf as workaround for the failure on building the code in Github actions
ENV ERL_FLAGS="+JPperf true"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

COPY lib lib

COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release --path /app_release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && \
    apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

ARG MIX_ENV=prod

WORKDIR /deploy/
ENV HOME=/deploy
ENV MIX_ENV=${MIX_ENV}

ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8

# # Copying source code to the destination
COPY --from=builder --chown=nobody:root /app_release/ ./


# # This will be updated to a random user
USER nobody
EXPOSE 4000
# If using an environment that doesn't automatically reap zombie processes, it is
# advised to add an init process such as tini via `apt-get install`
# above and adding an entrypoint. See https://github.com/krallin/tini for details
# ENTRYPOINT ["/tini", "--"]

CMD ["/deploy/bin/server"]

I did some changes to the Dockerfile to make it better:

I create the release in an specific path. That allows to make everything generic as it does not depend on the version or name of the app.
I execute it using server, exposing the port 4000, instead of application. It is an alias and works better

My question: I don’t need the JPperf parameter to make it run locally. What does it do? Why is not there by default? The documentation states that it provides profiling, and I am not sure why is needed in this case or what it changes to make the container work.

sucipto · November 15, 2024, 7:49am

Github action cross build for arm64 still use QEMU, and it’s QEMU bug:

maybe wait until github action support arm64 natively.

garazdawi · November 15, 2024, 8:23am

I don’t need the JPperf parameter to make it run locally. What does it do? Why is not there by default? The documentation states that it provides profiling, and I am not sure why is needed in this case or what it changes to make the container work.

+JPperf true enables profiling, but it also changes the native code generation to use single page mapping instead of dual. As of Erlang/OTP 26 you can force usage of single page mapping by using +JMsingle true. Using dual page mapping is a security measure agaist possible expoits of bugs in in the JIT, so you normally want this, but as @sucipto sais, qemu has bugs when running using dual mapping so you need to disable it there. From what I can tell, QEMU 8.1 or later should have the bug fixed.