How to copy dependencies from outside Docker image build?

Have you tried more advanced Dockerfile syntax?

Slides 13 an on,
RUN --mount=type=cache

This line is a bit suspicious:

COPY _build _mix deps mix.exs mix.lock /app/

Are you using the gitlab’s CI cache? And if yes, where this cache is populated in the first place?

I would suggest to drop the _build cache if you compiled these files anywhere else before, because I also had this problem, I ended up dropping cache on “docker in docker” builds since I had some cryptic compile errors.

That might be faster in this case, but is there any reason why it shouldn’t work with plain old copy?

Yeah, I have a previous step looking like this:

.marsvin:
  image: hexpm/elixir:1.15.4-erlang-26.0.2-debian-bookworm-20230612-slim
  variables:
    MIX_ENV: "test"
    # To have hex and rebar included in cache/artifacts, put MIX_HOME inside
    # the build root.
    MIX_HOME: "${CI_PROJECT_DIR}/packages/marsvin/_mix"

marsvin_mix_install:
  extends: .marsvin
  stage: init
  needs: []
  dependencies: []
  cache:
    key: "marsvin_mix_install"
    paths:
      - packages/marsvin/_mix
      - packages/marsvin/_build
      - packages/marsvin/deps
  before_script:
    - cd packages/marsvin
    - mix local.hex --force --if-missing
    - mix local.rebar --force --if-missing
  script:
    - mix deps.get
    - mix deps.clean --unused
    - mix compile --warnings-as-errors
  artifacts:
    paths:
      - packages/marsvin/_mix
      - packages/marsvin/_build
      - packages/marsvin/deps
    expire_in: 7 days

The exported artifacts are then used in the build step, like this:

marsvin_build:
  extends: .marsvin
  stage: build
  needs:
    - "marsvin_mix_install"
    - "marsvin_mix_test"
  dependencies: ["marsvin_mix_install"]
  image: ${BUILDKIT_IMAGE}
  variables:
    APP_VERSION: "${APP_VERSION}"
    IMAGE_URL_INTERNAL: "${DOCKER_REGISTRY_INTERNAL}/${DOCKER_NS_INT}/marsvin"
  script:
    - cd packages/marsvin
    - ${BUILDKIT_SCRIPT}
      --tag=${IMAGE_URL_INTERNAL}:${IMAGE_VERSION}
      --cache-repo=${IMAGE_URL_INTERNAL}:cache
      --context=.
      --file=Dockerfile

Yeah, I only added that to try to solve aforementioned problem, does not appear to make a difference anyway.

This solves the external cache problem, but I don’t know why it is failing. Are you using the same user to run your container?

Why don’t you use a release for this? Then you can have a multistage Dockerfile, and keep the running container lean (you don’t need Elixir there, just the libraries and the minimal requirements needed to run the release. Actually, mix phx.gen.release uses that approach and works really well (an the final image is smaller and does not contain anything not needed so it is more secure and faster to copy)

True, it might be (much) easier to simply build the release outside of the Docker-in-Docker environment, and reduce the Dockerfile to just copy that and call it done. That neatly sidesteps the dependency problem. I’ll try that instead.

Edit: Building the release outside the Dockerfile works great. Thanks for the idea.

It’s probably not worth mentioning now but your cache key being static would mean every build that uses marsvin_mix_install would try to share the same cache. That’s every commit on every branch.

I use the variable $CI_COMMIT_REF_NAME which creates a cache for each branch. A good visual primer is A visual guide to GitLab CI/CD caching to understand the various strategies. A single key may be fine if you don’t branch at all since I think later commits drop earlier jobs.

You can also do print debugging by adding RUN ls -al cache/directory to output contents in the logs. Its hacky as hell and there may be better approaches like setting up a runner locally. You could also do stuff like that in the .gitlab-ci.yml script or before script sections.

Just reviewing your .gitlab-ci.yml. In marsvin_build you have both needs: and dependencies: . This might work just fine, but Gitlab recommends not having both in the same layer: CI/CD YAML syntax reference | GitLab Docs, as one relies on current stage, and the other relies on previous build stage. IDK if that changes anything, but thanks for sharing your scripts.