Why does running `mix ecto.create` trigger a new compilation?

itopiz

In the process of setting up a Gitlab CI pipeline, I noticed something that I don’t understand.

My .gitlab-ci.yml looks like this:

variables:
  POSTGRES_HOST: "postgres"
  POSTGRES_DB: "postgres"
  POSTGRES_USER: "postgres"
  POSTGRES_PASSWORD: "postgres"
  MIX_ENV: "test"
  
stages:
  - build
  - test

default:
  image: elixir:1.9
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

compile:
  stage: build
  script:
    - apt-get update
    - apt-get install -y postgresql-client
    - mix deps.get --only test
    - mix compile --warnings-as-errors
  artifacts:
    paths:
      - _build
      - deps
    expire_in: 1 week

lint:
  stage: test
  script:
    - mix format --check-formatted

test:
  stage: test
  services:
    - postgres:10.10
  script:
    - mix event_store.setup
    - mix ecto.create
    - mix ecto.migrate
    - mix test

Since jobs are run independently from each other, I save the compilation artifacts from the compile job to avoid recompiling in the next jobs. However, when the test job runs the mix ecto.create command, a compilation is triggered again. Why? Did not the compile job compile everything needed when running mix compile --warnings-as-errors?

7 comments

/phoenix #mix #ecto-migration #gitlab

0 2635 7

2019-08-22 09:47:50 UTC

Showing Posts 1 to 7

Show Best Posts
Show All (oldest first)
Show All (newest first)

LostKobrakai

Each mix env does have it’s own artefacts and mix test defaults to running in the test mix env, while other mix tasks, which don’t have a preferred env, default to dev.

Post #1

itopiz

Thank you for your quick reply LostKobrakai.

All commands are run for the test environment thanks to the variable in .gitlab-ci.yml

variables:
  MIX_ENV: "test"

Also, browsing the artifacts with Gitlab shows that there is only a test directory within the _build directory.

So I guess all mix tasks were run in the test env, right?

Post #2

axelson

Scenic Core Team

If the jobs are run independently are the clocks synchronized? Maybe the timestamps are out of sync which might trigger the recompilation

Post #3

NobbZ

You do not “depend” on the previous stage, so there are no artifacts available during test stage.

Post #4

itopiz

Thank you for your answer. The Gitlab documentation about dependencies states:

Note that artifacts from all previous stages are passed by default.

So the artifacts should be passed. Also, the test job trace shows that the artifacts were downloaded:

Running with gitlab-runner 12.1.0 (de7731dd)
  on docker-auto-scale ed2dce3a
Using Docker executor with image elixir:1.9 ...
Starting service postgres:10.10 ...
Pulling docker image postgres:10.10 ...
Using docker image sha256:cf7301675e300ca20b74fb7fc3e34d2bfb04c6bc79699594217b0060e2312bcd for postgres:10.10 ...
Waiting for services to be up and running...
Pulling docker image elixir:1.9 ...
Using docker image sha256:f14e6364d0696d81769b31d6fb38f24b3bf10bf2503b005e5753cb6b10d746a4 for elixir:1.9 ...
Running on runner-ed2dce3a-project-13900485-concurrent-0 via runner-ed2dce3a-srm-1566462193-44c7355f...
Fetching changes...
Initialized empty Git repository in /builds/me/myapp/.git/
Created fresh repository.
From https://gitlab.com/me/myapp
 * [new branch]      master            -> origin/master
 * [new branch]      setup-ci-pipeline -> origin/setup-ci-pipeline
Checking out a7f07670 as setup-ci-pipeline...

Skipping Git submodules setup
Downloading artifacts for compile (277662661)...
Downloading artifacts from coordinator... ok        id=277662661 responseStatus=200 OK token=MDDAoYgU
$ mix local.hex --force
* creating /root/.mix/archives/hex-0.20.1
$ mix local.rebar --force
* creating /root/.mix/rebar
* creating /root/.mix/rebar3
$ mix event_store.setup
The EventStore database has been created.
The EventStore database has been initialized.
$ mix ecto.setup
Compiling 356 files (.ex)
Compiling lib/myapp_web/resolvers/accounts/change_user_role.ex (it's taking more than 15s)
[…]
Generated myapp app
The database for myapp.Repo has been created
The database for myapp.ReadStore has been created
$ mix test
Excluding tags: [:pending]

......................................................................... […]

Finished in 207.5 seconds
9 doctests, 3454 tests, 0 failures

Randomized with seed 555244
Job succeeded

Post #5

NobbZ

I’ve never checked for that. I always explicitely define dependencies, to make sure to have the artifacts I want.

Anyway, mix does solely rely on the files timestamps when deciding wether or not a file/dependency needs recompilation.

The artifacts might be extracted and carry the original date, while git on a checkout always sets current time as cdate.

This is an assumption though. I usually do not care for rebuilding such stuff on subsequent stages (in elixir), as I have to recompile anyway for different environments…

You might try to recursively touch all artifacts, alternatively some mix tasks allow --no-compile or something like that, you might be able to play with that.

Post #6