Elixir docker image won't build for linux/arm64/v8 using Github Actions

I just updated my docker file yesterday and running it with GitHub actions - these images worked:

ARG BUILDER_IMAGE="hexpm/elixir:1.14.3-erlang-25.3-debian-bullseye-20230227-slim"
ARG RUNNER_IMAGE="debian:bullseye-20230522-slim"

I see Github Actions logs aren’t public, I’m adding relevant log snippet below

[1]

2023-06-08T15:55:17.1581083Z #9 [4/9] RUN mix local.hex --force
2023-06-08T15:55:22.9893589Z #9 5.937 
2023-06-08T15:55:22.9894317Z #9 5.937 15:55:22.879 [notice] Application ssl exited: :ssl_app.start(:normal, []) returned an error: an exception was raised:
2023-06-08T15:55:22.9894919Z #9 5.937     ** (FunctionClauseError) no function clause matching in :gen.where/1
2023-06-08T15:55:22.9896120Z #9 5.937         (stdlib 5.0) gen.erl:615: :gen.where(KV.MixProject)
2023-06-08T15:55:22.9896569Z #9 5.937         (stdlib 5.0) gen.erl:107: :gen.start/6
2023-06-08T15:55:22.9897006Z #9 5.937         (ssl 11.0) ssl_sup.erl:46: :ssl_sup.init/1
2023-06-08T15:55:22.9897464Z #9 5.937         (stdlib 5.0) supervisor.erl:330: :supervisor.init/1
2023-06-08T15:55:22.9898145Z #9 5.937         (stdlib 5.0) gen_server.erl:962: :gen_server.init_it/2
2023-06-08T15:55:22.9898616Z #9 5.937         (stdlib 5.0) gen_server.erl:917: :gen_server.init_it/6
2023-06-08T15:55:22.9899094Z #9 5.937         (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0788015Z #9 5.942 
2023-06-08T15:55:23.0788731Z #9 5.942 15:55:22.983 [notice] Application public_key exited: :stopped
2023-06-08T15:55:23.0790572Z #9 5.942 
2023-06-08T15:55:23.0791032Z #9 5.942 15:55:22.987 [notice] Application asn1 exited: :stopped
2023-06-08T15:55:23.0791426Z #9 5.942 
2023-06-08T15:55:23.0791815Z #9 5.942 15:55:22.988 [notice] Application crypto exited: :stopped
2023-06-08T15:55:23.0874175Z #9 5.976 
2023-06-08T15:55:23.0874905Z #9 5.976 15:55:22.989 [error] Task #PID<0.116.0> started from #PID<0.101.0> terminating
2023-06-08T15:55:23.0876893Z #9 5.976 ** (MatchError) no match of right hand side value: {:error, {:ssl, {{:function_clause, [{:gen, :where, [KV.MixProject], [file: 'gen.erl', line: 615]}, {:gen, :start, 6, [file: 'gen.erl', line: 107]}, {:ssl_sup, :init, 1, [file: 'ssl_sup.erl', line: 46]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 330]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 962]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 917]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 241]}]}, {:ssl_app, :start, [:normal, []]}}}}
2023-06-08T15:55:23.0877741Z #9 5.976     (mix 1.14.5) lib/mix/utils.ex:647: Mix.Utils.read_httpc/1
2023-06-08T15:55:23.0878103Z #9 5.976     (mix 1.14.5) lib/mix/utils.ex:579: anonymous fn/2 in Mix.Utils.read_path/2
2023-06-08T15:55:23.0878508Z #9 5.976     (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
2023-06-08T15:55:23.0878894Z #9 5.976     (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
2023-06-08T15:55:23.0880131Z #9 5.976     (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0880484Z #9 5.976 Function: #Function<5.9853827/0 in Mix.Utils.read_path/2>
2023-06-08T15:55:23.0880753Z #9 5.976     Args: []
2023-06-08T15:55:23.0881027Z #9 5.983 ** (EXIT from #PID<0.101.0>) an exception was raised:
2023-06-08T15:55:23.0882506Z #9 5.983     ** (MatchError) no match of right hand side value: {:error, {:ssl, {{:function_clause, [{:gen, :where, [KV.MixProject], [file: 'gen.erl', line: 615]}, {:gen, :start, 6, [file: 'gen.erl', line: 107]}, {:ssl_sup, :init, 1, [file: 'ssl_sup.erl', line: 46]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 330]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 962]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 917]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 241]}]}, {:ssl_app, :start, [:normal, []]}}}}
2023-06-08T15:55:23.0883474Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:647: Mix.Utils.read_httpc/1
2023-06-08T15:55:23.0883970Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:579: anonymous fn/2 in Mix.Utils.read_path/2
2023-06-08T15:55:23.0884365Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
2023-06-08T15:55:23.0884757Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
2023-06-08T15:55:23.0885123Z #9 5.983         (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0885391Z #9 5.983 
2023-06-08T15:55:23.0887145Z #9 ERROR: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 1
2023-06-08T15:55:23.0887681Z ------
2023-06-08T15:55:23.0887965Z  > [4/9] RUN mix local.hex --force:
2023-06-08T15:55:23.0889362Z #9 5.976 Function: #Function<5.9853827/0 in Mix.Utils.read_path/2>
2023-06-08T15:55:23.0889653Z #9 5.976     Args: []
2023-06-08T15:55:23.0889930Z #9 5.983 ** (EXIT from #PID<0.101.0>) an exception was raised:
2023-06-08T15:55:23.0891532Z #9 5.983     ** (MatchError) no match of right hand side value: {:error, {:ssl, {{:function_clause, [{:gen, :where, [KV.MixProject], [file: 'gen.erl', line: 615]}, {:gen, :start, 6, [file: 'gen.erl', line: 107]}, {:ssl_sup, :init, 1, [file: 'ssl_sup.erl', line: 46]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 330]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 962]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 917]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 241]}]}, {:ssl_app, :start, [:normal, []]}}}}
2023-06-08T15:55:23.0892679Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:647: Mix.Utils.read_httpc/1
2023-06-08T15:55:23.0893025Z #9 5.983         (mix 1.14.5) lib/mix/utils.ex:579: anonymous fn/2 in Mix.Utils.read_path/2
2023-06-08T15:55:23.0893426Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
2023-06-08T15:55:23.0893821Z #9 5.983         (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
2023-06-08T15:55:23.0894189Z #9 5.983         (stdlib 5.0) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
2023-06-08T15:55:23.0894456Z #9 5.983 
2023-06-08T15:55:23.0894824Z ------
2023-06-08T15:55:23.0895011Z Dockerfile:7
2023-06-08T15:55:23.0895265Z --------------------
2023-06-08T15:55:23.0895699Z    5 |     ENV MIX_ENV=prod
2023-06-08T15:55:23.0895922Z    6 |     
2023-06-08T15:55:23.0896328Z    7 | >>> RUN mix local.hex --force
2023-06-08T15:55:23.0896653Z    8 |     RUN mix local.rebar --force
2023-06-08T15:55:23.0896882Z    9 |     
2023-06-08T15:55:23.0897247Z --------------------
2023-06-08T15:55:23.0897728Z ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 1
2023-06-08T15:55:23.0992749Z ##[error]buildx failed with: ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 1

[2]

2023-06-08T16:00:31.0964557Z #9 [4/9] RUN mix local.hex --force
2023-06-08T16:00:32.5990708Z #9 1.498 Segmentation fault (core dumped)
2023-06-08T16:00:32.6999995Z #9 ERROR: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 139
2023-06-08T16:00:32.7000459Z ------
2023-06-08T16:00:32.7000840Z  > [4/9] RUN mix local.hex --force:
2023-06-08T16:00:32.7001115Z #9 1.498 Segmentation fault (core dumped)
2023-06-08T16:00:32.7001384Z ------
2023-06-08T16:00:32.7001593Z Dockerfile:7
2023-06-08T16:00:32.7001851Z --------------------
2023-06-08T16:00:32.7002127Z    5 |     ENV MIX_ENV=prod
2023-06-08T16:00:32.7002334Z    6 |     
2023-06-08T16:00:32.7002616Z    7 | >>> RUN mix local.hex --force
2023-06-08T16:00:32.7002950Z    8 |     RUN mix local.rebar --force
2023-06-08T16:00:32.7003190Z    9 |     
2023-06-08T16:00:32.7003436Z --------------------
2023-06-08T16:00:32.7004241Z ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 139
2023-06-08T16:00:32.7127424Z ##[error]buildx failed with: ERROR: failed to solve: process "/bin/sh -c mix local.hex --force" did not complete successfully: exit code: 139

It looks like this is caused by QEMU not liking our JIT.

Thanks for the suggestion! I tried using the builder image you posted, but it still fails . Were you able to make it work on the architecture linux/arm64/v8?

Thank you for finding this. Do you know if there is an alternative for QEMU or some other solution for the problem or could it be that everyone who uses Github Actions is doomed?

I’ll see if I can make a PR that fixes this in QEMU, it doesn’t seem to be a very high priority for them.

For the time being you can disable the JIT by compiling Erlang with ./configure --disable-jit.

I am absolutely not as well versed as an Erlang core contributor but as a blind shot in the dark I’d also try the official hexpm Docker images. There’s a ton of Erlang and Elixir version combinations.

What about disabling jit for building… lets hope there a flag
Edit: little to trigger happy now I see @jhogberg advice

I’ll give building my own image a try. If you find the time for making the PR with the fix, that would be amazing!

At the end, I stopped using QEMU and started building images with BuildJet for GitHub Actions | BuildJet for GitHub Actions.

It works, it’s faster, and cheaper than using GitHub hosted runners.

Thanks for help everyone!