jgallinari
Apple silicon and cross-platform Docker fails Minikube
Hello,
I made the switch recently from an x86 MacBook to Apple silicon (M3) and it seems to fail my Docker/Minikube workflow that perfectly worked under my old machine.
My app is a Phoenix app that is started in Minikube with a sidecar container (same pod) doing the migrations before the real app is started.
I’m using Elixir v1.15.7 with OTP v26.
FYI, the migrations are run in Dockerfile via
command: ['/home/elixir/app/bin/oss', 'eval', 'Oss.Migrator.migrate']
First thing, I found out there is no way to Docker build the app unless I add this ERL flag in the Dockerfile (as pointed out by Mix deps.get memory explosion when doing cross-platform Docker build]):
ENV ERL_FLAGS="+JPperf true"
or
ENV ERL_FLAGS="+JMsingle true"
I also had to use docker build --platform linux/amd64 to make it work in Minikube.
Then the rest of it:
- Run Docker Desktop.
- Start Minikube with
minikube start --extra-config=apiserver.service-node-port-range=1-50000 --memory=6g --cpus=4 --kubernetes-version=v1.22.15 --cni=calico --driver=docker
-
Docker build
-
Run the app in Minikube (via terraform apply), which causes this weird error while doing the migrations:
** (ArgumentError) could not call Module.put_attribute/3 because the module Oss.Repo.Migrations.CreatePopsTable is already compiled │
│ (elixir 1.15.7) lib/module.ex:2310: Module.assert_not_readonly!/2 │
│ (elixir 1.15.7) lib/module.ex:2007: Module.__put_attribute__/5 │
│ lib/oss-1.14.0/priv/repo/migrations/20210000000300_create_pois_table.exs:2: (module)
This error makes absolutely no sense to me, especially because it does not occur with my x86 MacBook.
If you think of anything I could try to sort this out, I’d be grateful for that
.
Marked As Solved
Tyson
Okay, this feels like the real problem. Is it possible that there is a base image further down the stack which is x86? I believe this can come up when a base image doesn’t offer an ARM version.
Also Liked
jhogberg
Cross-architecture support in docker is iffy at best since it uses QEMU in user-mode emulation mode, which:
- Cannot handle JIT’ed code gracefully in all cases, as it is unaware of dual-mapped pages and cannot detect that code mapped that way has been modified. The
+JMsingle trueflag (implied by+JPperf true) works around the most egregious bugs, but not all. - Keeps multi-threaded code generation enabled even when this is known not to work due to differences in the target and host memory orders. Emulating ARM (weakly ordered) on x86 (strongly ordered) is fine, but the opposite is a recipe for disaster.
In short, don’t use cross-architecture docker unless you want to spice up your life with super-mysterious bugs.
Does the problem happen outside of a container? Does it work when disabling the JIT (--disable-jit configure flag)?
jgallinari
Yes, the hexpm/elixir base image that I was using is x86 based.
Once I could try it with the linux/arm64 version of the image, the problem was gone, can you believe it? ![]()
Thanks a lot, I really appreciate your support!
[edit: and no more need for the ERL_FLAGS, as you said]
Tyson
I’ve had good luck building/running x86 images from ARM host by configuring the base Erlang image to --disable-jit.
I haven’t yet tried the ERL_FLAGS approach that was giving jgallinari trouble here.








