How to avoid stale overlay files in mix releases

Questions / Help
1

I noticed a weird behavior in one of the CI/CD pipeline I’m working on: I removed a file inside rel/overlays, but the CI included it inside the mix release regardless.

I’ve looked into it and found out that the release stored inside _build/$env/rel/$name has all the stale files copied from old overlays and I can’t get rid of those files even when I run mix release --overwrite --force, I’ll paste the steps to reproduce this issue in an empty project:

mix new my_project
cd my_project

# I'm just adding a release to the mix.exs
cat << EOF > mix.exs
defmodule MyProject.MixProject do
  use Mix.Project
  def project do
    [
      app: :my_project,
      version: "0.1.0",
      elixir: "~> 1.15",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      releases: [
        demo: [
          include_executables_for: [:unix],
          applications: [runtime_tools: :permanent]
        ]
      ]
    ]
  end
  def application, do: [ extra_applications: [:logger] ]
  defp deps, do: []
end
EOF

mkdir -p rel/overlays

# create two files in `rel/overlays`
touch rel/overlays/file1 rel/overlays/file2
mix release
ls _build/dev/rel/demo # both file1 and file2 are there

rm rel/overlays/file2
mix release --overwrite --force
ls _build/dev/rel/demo # `file2` is still there

Obviously if I remove _build/dev/rel the next release will only include file1, as expected. The problem is that our pipeline caches both deps and _build by default, so the only solution was to remove the whole cache and rebuild everything from scratch.

I’m trying to find a solution to avoid this issue in the future, my first idea was to cache only _build/$MIX_ENV/lib which I guess would work but that cache would not include dialyzer’s PLTs.
My second guess was to run rm -rf _build/$MIX_ENV/rel before running mix release --overwrite which I think would also work, but at this point I’m wondering if I’m missing something.

So my questions are: how do you cache your _build in your CI? How do you avoid stale files in you releases?

2

I use robocopy for something similar so maybe you could rsync the overlays folder?

3

Are you sure this behavior is consistent after removing _build folder the first time?

The caching system can sometimes bug out, but this happens rarely and usually it is very hard to make the bug results reproducible.

Personally for building the release I disable the cache, takes a few more minutes, however beats having to deal with these kind of problems.

4

Yeah, I’ve tried multiple times and even with different elixir versions and inside a docker container, apparently the normal mix release behavior is to copy the overlays to the release directory without removing the old files before.

I also have some projects where the cache is disabled for the release to avoid caching issues, but this specific behavior is very surprising and maybe we could add something like File.rm_rf!(release.version_path) in release.exs before run_steps/1 so it actually overwrites the old release?

EDIT: I just realized it already happens here, but this just deletes _build/$env/rel/$name/releases/$version, so it leaves all the files from the overalys there