Apple silicon and cross-platform Docker fails Minikube

I can’t speak to your specific migration error, but I’m surprised that you need to target x86 to build/run a Docker image in your local environment.

I’ve had to wrestle with cross-arch Elixir+Docker quirks when building images on my M1 Macbook to run on x86 hosts, but have had no issues building ARM images to run locally on my Macbook.

Tho I’m also using the k8s “cluster” built into Docker Desktop instead of minikube.

You’re absolutely right, I actually don’t need to build my Docker image with --platform linux/amd64.
I can now reproduce the same error with an arm64 image .

Cross-architecture support in docker is iffy at best since it uses QEMU in user-mode emulation mode, which:

• Cannot handle JIT’ed code gracefully in all cases, as it is unaware of dual-mapped pages and cannot detect that code mapped that way has been modified. The +JMsingle true flag (implied by +JPperf true) works around the most egregious bugs, but not all.
• Keeps multi-threaded code generation enabled even when this is known not to work due to differences in the target and host memory orders. Emulating ARM (weakly ordered) on x86 (strongly ordered) is fine, but the opposite is a recipe for disaster.

In short, don’t use cross-architecture docker unless you want to spice up your life with super-mysterious bugs.

Does the problem happen outside of a container? Does it work when disabling the JIT (--disable-jit configure flag)?

Seems like other folks who have run into this were inadvertently using erlang/elixir installations (or macOS system libs) originally compiled for x86 and/or have Rosetta emulation enabled in their terminal. When you switched to your M3 Macbook, did you use Apple’s migration tool to move things over from your Intel machine? Could be some lingering x86 packages from that.

I believe @jhogberg is referring to using the --disable-jit flag when configuring the Erlang build, which I have had good luck with when building cross-arch, but if you’re now using an ARM Docker image, I don’t think that’s the issue here.

[edit: I believe Rosetta emulation only comes into play while installing elixir/erlang packages, as it may result in x86 versions being installed rather than native ARM]

[edit: my suggestions about macOS sys libs is probably irrelevant to your Docker build ]

Not sure where to add --disable-jit in my Dockerfile, I tried

ENV KERL_CONFIGURE_OPTIONS="--disable-jit"

with not much effect in the error happening.

To answer your questions:

• No, the problem does not happen outside of a container.
• And no, I did not use Apple’s migration tool because I feared it could cause some x86 vs arm issues

In addition, I also did some trials by replacing Docker Desktop with Colima run like this:

colima start --memory 6 --cpu 4 --dns 100.100.100.100 --arch aarch64 --vm-type=vz --vz-rosetta

which required adding the --base-image gcr.io/k8s-minikube/kicbase:v0.0.40 option to minikube start to make it work, like reported in Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images · Issue #17700 · kubernetes/minikube · GitHub because Minikube v1.32.0 which is based on kicbase:v0.0.42 causes issues with Colima (not only OrbStack as mentioned in the Minikube issue).

That to say that since I use the Apple Virtualization Framework + Rosetta enabled in Colima, QEMU is not in use in my workflow.
But using Colima stills results in the exact same error

I’m still struggling with this part:

I’ve never needed to do any extra config to build ARM images to run on my local ARM machine. I’ve only had to mess with that while building on ARM for x86 platform. Are you sure the Elixir base image you’re using is also ARM? For instance:

$ docker run --rm elixir:1.15-alpine uname -m
aarch64

But in case this is helpful (and maybe there’s a more elegant way to do this), you can copy the Erlang Dockerfile and update it to read:

...
    && ./configure --build="$gnuArch" --disable-jit \
...

…then build the Erlang image, then create a local copy of the Elixir Dockerfile to build FROM ... that local Erlang image in order to create your own JIT-disabled Elixir base image.

One more thing I’ll add: in some circumstances I’ve found it necessary to specify the platform in the Dockerfile (rather than with the docker build flag):

FROM --platform=linux/arm64 <IMAGE_NAME>

Not sure why that sometimes makes a difference.

That’s the output I get indeed.

If I don’t use

ENV ERL_FLAGS="+JMsingle true"

in the Dockerfile, then

• if I use Docker Desktop, I get

 => ERROR [build  5/10] RUN mix deps.get --only prod                                                                                                                                                             1.0s
------
 > [build  5/10] RUN mix deps.get --only prod:
1.000
1.000 21:33:37.562 [notice] Application ssl exited: exited in: :ssl_app.start(:normal, [])
1.000     ** (EXIT) an exception was raised:
1.000         ** (MatchError) no match of right hand side value: []
1.000             (elixir 1.15.7) src/elixir_def.erl:134: :elixir_def.store_definition/3
1.000             (ssl 11.0.3) ssl_app.erl:47: :ssl_app.start_logger/0
1.000             (ssl 11.0.3) ssl_app.erl:32: :ssl_app.start/2
1.000             (kernel 9.1) application_master.erl:293: :application_master.start_it_old/4

• and if I use Colima, it seemingly hangs forever on the same step.

And this I get while calling docker build without specifying a plaftorm, but using FROM --platform=linux/amd64 <custom-elixir-alpine-image>

Why specify AMD platform for something you’re going to run locally on ARM?

This doesn’t really sound like an issue with the architecture. Sounds like :ssl_app.start(:normal, []) is not a valid invocation with empty list.

[edit: but I guess you’re not calling that, so could be some cross-arch fallout]