I’ve been working on many projects based on Phoenix in the last few years and worked on the transition from mix phx.server (for production environment) to OTP release with distillery then mix release. Starting last night, for a reason I can not explain, splitting mix deps.get and mix compile on different docker intermediate layers does not work anymore :
COPY mix.* ./
RUN mix deps.get --only ${MIX_ENV}
COPY . .
RUN mix compile
And fails with the following errors:
Unchecked dependencies for environment prod:
* parse_trans (Hex package)
lock mismatch: the dependency is out of date. To fetch locked version run "mix deps.get"
* mimerl (Hex package)
lock mismatch: the dependency is out of date. To fetch locked version run "mix deps.get"
…
* phoenix_ecto (Hex package)
lock mismatch: the dependency is out of date. To fetch locked version run "mix deps.get"
** (Mix) Can't continue due to errors on dependencies
The command '/bin/sh -c mix compile' returned a non-zero code: 1
Moving RUN mix deps.get after COPY . . works, but it defies the caching purposes of isolating dependencies… I tried on both MacOS and Ubuntu 18.04 with the same result
We have been using this setup for as long as I can remember for every ruby/node/elixir projects we are packaging as Docker images. Take a look at the elixir-boilerplate project we are maintaining to start every web projects at Mirego.
There was an update to hex last night, there were some other errors prior to the update around parsing the old lock format. I saw similar errors to yours after the update, but a rerun resolved them.
I’m pretty sure it has to do with the docker caching, so you may need to break the cache.
I had to mix local.hex to update hex locally, then mix deps.get to update the lock file to the “fixed” format, then my docker build started working again!
> mix local.hex
Found existing entry: /Users/gcauchon/.asdf/installs/elixir/1.9.4-otp-22/.mix/archives/hex-0.20.1
Are you sure you want to replace it with "https://repo.hex.pm/installs/1.9.0/hex-0.20.4.ez"? [Yn] y
* creating /Users/gcauchon/.asdf/installs/elixir/1.9.4-otp-22/.mix/archives/hex-0.20.4
For future references to anyone who might come across the same issue, here are the related issues:
If anyone is having Hex+Docker-related failures on CircleCI in particular, note that their layer caching is non-deterministic and can’t be intentionally cache-busted, so you’ll have to turn it off altogether for about a week:
This happens when you have a manifest file that is newer than your mix.lock file. That’s because you run mix deps.get which updates your manifest and lock file but then your dockefile’s COPY command introduces the old lockfile again. I would recommend against doing COPY . . in your dockerfile, you should instead only copy what is needed for the next build step.
I will publish a new release of Hex that mitigates this issue.
I haven’t deployed anything yet, but the lock file issue usually happens to me in development (because my volume mounted lock file has the old lock file). Running a docker-compose run web mix deps.get fixes it and runs almost instantly. That gets the newly built lock file from the running container back to the Docker host.
But I’m curious how that would happen in a CI or production environment because typically you wouldn’t be using a volume mount in those cases.
It’s an internal manifest to Hex we write to deps/$DEP/.hex with information about the fetched dependency. We diff the information in the manifest file against the lockfile to determine if the the local dependency is outdated and needs to be fetched again to match the lock.
In this case we marked the dependency as outdated because the manifest had a new field :outer_checksum that was missing in the lock.
It happens when you do COPY . . after RUN mix deps.get, because it copies everything from the host to the container, including mix.lock which is potentially outdated compared to the manifest in the container.
Now all of the dependencies are outside the scope of where you would COPY . . and you don’t need to worry about all of your deps leaking back to your volume in development too.
I do the same thing with Yarn too for node_modules/.
Means that the mix.lock file will now contain a checksum of all dependencies? @ericmj I am not criticising, simply trying to understand. I have to justify a rather big diff to our project’s mix.lock file and I am looking for the exact argumentation to present.
We have started using a new checksum for Hex packages that plugs a possible vulnerability with the old checksum. Because of backwards compatibility we cannot replace the old checksum so instead we add a new field with the new checksum.
In the past we have only added new fields when the dependency changed (either upgrade or downgrade) but since this new checksum was added for security reasons we decided to always add it after dependency resolution.
Thanks a lot, this both improves my understanding and it gives me an official answer to help push through our local bureaucracy so a big mix.lock diff PR gets approved.
I did want to offer one additional piece (didn’t see it in the thread), but in your phoenix folder, add a .dockerignore file and add the mix.lock to it, just so all the local version don’t get copied into the docker image.
I strongly recommend not ignoring mix.lock in your Docker context since this would have the unwanted side effect of unlocking your deps! The RUN mix deps.get step of your Docker build will download the latest version of every libs that match to the semver definition in your mix.exs; completely bypassing the purpose of the lock file!
Cheers to everyone involved in solving this issue today, the community is amazing…
I strongly recommend not ignoring 
