Elixir docker image won't build for linux/arm64/v8 using Github Actions

Questions / Help
,
1

I setup a sample project on Github to show you a concrete example of a problem I’m having.

When I use

the build fails. The same happens if I use Elixir 1.14.0, but a different error.

You can see errors on the links above.

If I use the same Dockerfile on Mac M2 or Raspberry PI (aarm64 architecture), the build succeeds.

I’m not sure what could be the issue.

The build also succeeds

  • if I use Github Actions but some other architecture.
  • if I use Elixir 1.13.4

Any idea would be much appreciated :pray:.

2

I just updated my docker file yesterday and running it with GitHub actions - these images worked:

ARG BUILDER_IMAGE="hexpm/elixir:1.14.3-erlang-25.3-debian-bullseye-20230227-slim"
ARG RUNNER_IMAGE="debian:bullseye-20230522-slim"
3

I see Github Actions logs aren’t public, I’m adding relevant log snippet below

[1]

2023-06-08T15:55:17.1581083Z #9 [4/9] RUN mix local.hex --force
2023-06-08T15:55:22.9893589Z #9 5.937 
2023-06-08T15:55:22.9894317Z #9 5.937 15:55:22.879 [notice] Application ssl exited: :ssl_app.start(:normal, []) returned an error: an exception was raised:
2023-06-08T15:55:22.9894919Z #9 5.937     ** (FunctionClauseError) no function clause matching in :gen.where/1
2023-06-08T15:55:22.9896120Z #9 5.937         (stdlib 5.0) gen.erl:615: :gen.where(KV.MixProject)
2023-06-08T15:55:22.9896569Z #9 5.937         (stdlib 5.0) gen.erl:107: :gen.start/6
2023-06-08T15:55:22.9897006Z #9 5.937         (ssl 11.0) ssl_sup.erl:46: :ssl_sup.init/1
2023-06-08T15:55:22.9897464Z #9 5.937         (stdlib 5.0) supervisor.erl:330: :supervisor.init/1
2023-06-08T15:55:22.9898145Z #9 5.937         (stdlib 5.0) gen_server.erl:962: :gen_server.init_it/2
2023-06-08T15:55:22.9898616Z #9 5.937         (stdlib 5.0) gen_server.erl:917: :gen_server.init_it/6
2023-06-08T15:55:22.9899094Z #9 5.937         (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0788015Z #9 5.942 
2023-06-08T15:55:23.0788731Z #9 5.942 15:55:22.983 [notice] Application public_key exited: :stopped
2023-06-08T15:55:23.0790572Z #9 5.942 
2023-06-08T15:55:23.0791032Z #9 5.942 15:55:22.987 [notice] Application asn1 exited: :stopped
2023-06-08T15:55:23.0791426Z #9 5.942 
2023-06-08T15:55:23.0791815Z #9 5.942 15:55:22.988 [notice] Application crypto exited: :stopped
2023-06-08T15:55:23.0874175Z #9 5.976 
2023-06-08T15:55:23.0874905Z #9 5.976 15:55:22.989 [error] Task #PID<0.116.0> started from #PID<0.101.0> terminating
2023-06-08T15:55:23.0876893Z #9 5.976 ** (MatchError) no match of right hand side value: {:error, {:ssl, {{:function_clause, [{:gen, :where, [KV.MixProject], [file: 'gen.erl', line: 615]}, {:gen, :start, 6, [file: 'gen.erl', line: 107]}, {:ssl_sup, :init, 1, [file: 'ssl_sup.erl', line: 46]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 330]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 962]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 917]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 241]}]}, {:ssl_app, :start, [:normal, []]}}}}
2023-06-08T15:55:23.0877741Z #9 5.976     (mix 1.14.5) lib/mix/utils.ex:647: Mix.Utils.read_httpc/1
2023-06-08T15:55:23.0878103Z #9 5.976     (mix 1.14.5) lib/mix/utils.ex:579: anonymous fn/2 in Mix.Utils.read_path/2
2023-06-08T15:55:23.0878508Z #9 5.976     (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
2023-06-08T15:55:23.0878894Z #9 5.976     (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
2023-06-08T15:55:23.0880131Z #9 5.976     (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0880484Z #9 5.976 Function: #Function<5.9853827/0 in Mix.Utils.read_path/2>
2023-06-08T15:55:23.0880753Z #9 5.976     Args: []
2023-06-08T15:55:23.0881027Z #9 5.983 ** (EXIT from #PID<0.101.0>) an exception was raised:
2023-06-08T15:55:23.0882506Z #9 5.983     ** (MatchError) no match of right hand side value: {:error, {:ssl, {{:function_clause, [{:gen, :where, [KV.MixProject], [file: 'gen.erl', line: 615]}, {:gen, :start, 6, [file: 'gen.erl', line: 107]}, {:ssl_sup, :init, 1, [file: 'ssl_sup.erl', line: 46]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 330]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 962]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 917]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 241]}]}, {:ssl_app, :start, [:normal, []]}}}}
2023-06-08T15:55:23.0883474Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:647: Mix.Utils.read_httpc/1
2023-06-08T15:55:23.0883970Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:579: anonymous fn/2 in Mix.Utils.read_path/2
2023-06-08T15:55:23.0884365Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
2023-06-08T15:55:23.0884757Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
2023-06-08T15:55:23.0885123Z #9 5.983         (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0885391Z #9 5.983 
2023-06-08T15:55:23.0887145Z #9 ERROR: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 1
2023-06-08T15:55:23.0887681Z ------
2023-06-08T15:55:23.0887965Z  > [4/9] RUN mix local.hex --force:
2023-06-08T15:55:23.0889362Z #9 5.976 Function: #Function<5.9853827/0 in Mix.Utils.read_path/2>
2023-06-08T15:55:23.0889653Z #9 5.976     Args: []
2023-06-08T15:55:23.0889930Z #9 5.983 ** (EXIT from #PID<0.101.0>) an exception was raised:
2023-06-08T15:55:23.0891532Z #9 5.983     ** (MatchError) no match of right hand side value: {:error, {:ssl, {{:function_clause, [{:gen, :where, [KV.MixProject], [file: 'gen.erl', line: 615]}, {:gen, :start, 6, [file: 'gen.erl', line: 107]}, {:ssl_sup, :init, 1, [file: 'ssl_sup.erl', line: 46]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 330]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 962]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 917]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 241]}]}, {:ssl_app, :start, [:normal, []]}}}}
2023-06-08T15:55:23.0892679Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:647: Mix.Utils.read_httpc/1
2023-06-08T15:55:23.0893025Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:579: anonymous fn/2 in Mix.Utils.read_path/2
2023-06-08T15:55:23.0893426Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
2023-06-08T15:55:23.0893821Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
2023-06-08T15:55:23.0894189Z #9 5.983         (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0894456Z #9 5.983 
2023-06-08T15:55:23.0894824Z ------
2023-06-08T15:55:23.0895011Z Dockerfile:7
2023-06-08T15:55:23.0895265Z --------------------
2023-06-08T15:55:23.0895699Z    5 |     ENV MIX_ENV=prod
2023-06-08T15:55:23.0895922Z    6 |     
2023-06-08T15:55:23.0896328Z    7 | >>> RUN mix local.hex --force
2023-06-08T15:55:23.0896653Z    8 |     RUN mix local.rebar --force
2023-06-08T15:55:23.0896882Z    9 |     
2023-06-08T15:55:23.0897247Z --------------------
2023-06-08T15:55:23.0897728Z ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 1
2023-06-08T15:55:23.0992749Z ##[error]buildx failed with: ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 1

[2]

2023-06-08T16:00:31.0964557Z #9 [4/9] RUN mix local.hex --force
2023-06-08T16:00:32.5990708Z #9 1.498 Segmentation fault (core dumped)
2023-06-08T16:00:32.6999995Z #9 ERROR: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 139
2023-06-08T16:00:32.7000459Z ------
2023-06-08T16:00:32.7000840Z  > [4/9] RUN mix local.hex --force:
2023-06-08T16:00:32.7001115Z #9 1.498 Segmentation fault (core dumped)
2023-06-08T16:00:32.7001384Z ------
2023-06-08T16:00:32.7001593Z Dockerfile:7
2023-06-08T16:00:32.7001851Z --------------------
2023-06-08T16:00:32.7002127Z    5 |     ENV MIX_ENV=prod
2023-06-08T16:00:32.7002334Z    6 |     
2023-06-08T16:00:32.7002616Z    7 | >>> RUN mix local.hex --force
2023-06-08T16:00:32.7002950Z    8 |     RUN mix local.rebar --force
2023-06-08T16:00:32.7003190Z    9 |     
2023-06-08T16:00:32.7003436Z --------------------
2023-06-08T16:00:32.7004241Z ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 139
2023-06-08T16:00:32.7127424Z ##[error]buildx failed with: ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 139
4

It looks like this is caused by QEMU not liking our JIT. :confused:

5

Thanks for the suggestion! I tried using the builder image you posted, but it still fails :thinking:. Were you able to make it work on the architecture linux/arm64/v8?

6

Thank you for finding this. Do you know if there is an alternative for QEMU or some other solution for the problem or could it be that everyone who uses Github Actions is doomed?

7

I’ll see if I can make a PR that fixes this in QEMU, it doesn’t seem to be a very high priority for them.

For the time being you can disable the JIT by compiling Erlang with ./configure --disable-jit.

2 Likes
8

I am absolutely not as well versed as an Erlang core contributor but as a blind shot in the dark I’d also try the official hexpm Docker images. There’s a ton of Erlang and Elixir version combinations.

9

What about disabling jit for building… lets hope there a flag
Edit: little to trigger happy now I see @jhogberg advice

10

I’ll give building my own image a try. If you find the time for making the PR with the fix, that would be amazing!

11

At the end, I stopped using QEMU and started building images with BuildJet for GitHub Actions | BuildJet for GitHub Actions.

It works, it’s faster, and cheaper than using GitHub hosted runners.

Thanks for help everyone!

12

As a small update, the patch for user-mode emulated AArch64 has been accepted and will most likely make it into the next major version of QEMU.

x86-64 is not covered by the patch, and may take much longer to get in if it’s accepted at all, as the workaround might not be as palatable (invalidating all translated code everywhere on serializing instructions like cpuid).

4 Likes
13

Every time I forget about this I end up reading this post and also this one: BEAM docker release GitHub action - a GitHub action to build a docker container from a BEAM (Elixir/Erlang) release - #2 by asabil - Libraries - Erlang Programming Language Forum - Erlang Forums

TL;DR, adding ERL_FLAGS="+JPperf true" to your env vars should solve the issue with QEMU when building for linux/arm64 on a linux/amd64 host.

3 Likes