How do I bring together automated tests and a layered application?

Starting with the simplest approach, would having a script that cd’s into each app and just runs mix test in each in turn not work?

I think it would, somehow. What we do in an older project, consisting of just one Phoenix application is, firing up a docker-in-docker image, building the container and running a shell script for the test inside.

The mix test in our output is gathered from the CI runner per stdout, so there is a lot of “noise” from the application container setup, compiler, yarn etc. The coverage dir is gathered as an artifact.

os, in theory it shoul be possible to extend the command script in docker to cd to every dir, run test and pipe output to a file, create the docs and coverage and export all as single artifact pahs (or split docs and tests into two pipeline jobs, triggered by tags, which may make more sense.

Initially I thought, there might be some other “best practice”, as a real “umbrella app” does these things implicitly through the global mix file, as far as I understood. So I would just trigger the tests in, for example, the phoenix application and a mix tasks would consecutively run tests in the pulled-in core and persistence dependencies.

But you are completely right, as a simple approach that would surely work. It would just (maybe) lead to some more effort in filtering out unwanted log noise.

Overall the solution we have come up with is to create a proper execution environment with Docker.

From the codebase we’d build 2 containers — 1 for deployment (running an Elixir release, configured to listen on a specific port etc) and 1 for testing (containing source code, MIX_ENV=test, able to run mix test, mix dialyzer, etc).

Then we write a Docker Compose file to bring up associated services such as PostgreSQL. Then it is a simple matter of running the desired command on TeamCity

I think the key is to recognise that you can build more than 1 Docker container from the same project and that if you can keep things simple (so running mix test from the root of the umbrella project tests everything) the solution is also simple.

Screenshot 2020-10-18 at 14.22.34969×391 48.9 KB

As to coveralls… it would seem that you actually have different kinds of tests — Unit Tests, Code Coverage Tests, etc. Don’t write any custom code to try and combine this into 1 test and 1 piece of output. Instead, leverage whatever CI/CD platform you have and run each kind of test separately.

Screenshot 2020-10-18 at 14.27.57207×230 8.82 KB

You can probably create some kind of dependencies between the different types of tests/checks/builds as well. Again, this depends on what your CI/CD platform is capable of. I would recommend taking a few days to properly research your options.

@evadne Thank you, that helped a lot. I did some more research and came up with a working pipeline for Gitlab CI. This surely is not the most elegant way to do things, but works out quite well for the moment.

Given my 3 mix projects “stackgate_interface” Phoenix, REST interface) depends on “stackgate_engine” (functional core and business logic) depends on “stackgate_persistence” (all db or file-related operations) I now do the following:

1. Compile engine and persistence and distribute deps and _build as artifacts
2. Compile interface and distribute deps and _build as artifacts
3. Import artifacts and check with mix format and Credo
4. Import artifacts and create new artifacts from mix coveralls --trace for engine and persistence
5. Same for interface, but this check has to integrate some end-to-end test with curl or postman, when done. So we delay tests
6. Import all available artifacts and output docs side by side with coverage report as new (final) artifact and let all others expire.
7. (Planned job) Build the release per multistage container and, as a parallel job, push image to registry,
8. (Planned stage) Deploy to staging and to prod after manual verification, upload docs to knowledge base and coverage/trace to archive.

For reference and hopefully as a little starter for others, I will put my current Gitlab CI configuration here.

This setup leads to the following pipeline with DAG configured for speeding up:

pipeline1387×311 22 KB

dag1054×472 35.6 KB

.gitlab-ci.yml:

stages:
  - build:engine
  - build:interface
  - lint
  - test:engine
  - test:interface
  - create:deployment
  - deploy

include:
  - local: "ci/build.yml"
  - local: "ci/lint.yml"
  - local: "ci/test.yml"
  - local: "ci/create.yml"
#  - local: "ci/deploy.yml"

variables:
  MIX_ENV: 'test'

ci/build.yml:

.elixir_default: &elixir_default
  image: wyrdforge/elixir:1.11.1-alpine
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

compile:persistence:
  <<: *elixir_default
  stage: build:engine
  script:
    - cd stackgate_persistence
    - mix deps.get --only-test
    - mix compile --warnings-as-errors
  artifacts:
    when: on_success
    expire_in: 1 hrs
    paths:
      - stackgate_persistence/_build
      - stackgate_persistence/deps

compile:engine:
  <<: *elixir_default
  stage: build:engine
  script:
    - cd stackgate_engine
    - mix deps.get --only-test
    - mix compile --warnings-as-errors
  artifacts:
    when: on_success
    expire_in: 1 hrs
    paths:
      - stackgate_engine/_build
      - stackgate_engine/deps

compile:interface:
  <<: *elixir_default
  stage: build:interface
  needs: ["compile:persistence", "compile:engine"]
  script:
    - cd stackgate_interface
    - mix deps.get --only-test
    - mix compile --warnings-as-errors
  artifacts:
    when: on_success
    expire_in: 1 hrs
    paths:
      - stackgate_interface/_build
      - stackgate_interface/deps

ci/lint.yml:

.elixir_default: &elixir_default
  image: wyrdforge/elixir:1.11.1-alpine
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

lint:persistence:
  <<: *elixir_default
  stage: lint
  needs: ["compile:persistence"]
  script:
    - cd stackgate_persistence
    - mix format --check-formatted
  dependencies:
    - compile:persistence
    
lint:engine:
  <<: *elixir_default
  stage: lint
  needs: ["compile:engine"]
  script:
    - cd stackgate_engine
    - mix format --check-formatted
    - mix credo --strict
  dependencies:
    - compile:engine

lint:interface:
  <<: *elixir_default
  stage: lint
  needs: ["compile:interface"]
  script:
    - cd stackgate_interface
    - mix format --check-formatted
  dependencies:
    - compile:interface

ci/test.yml:

.elixir_default: &elixir_default
  image: wyrdforge/elixir:1.11.1-alpine
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

test:persistence:
  <<: *elixir_default
  stage: test:engine
  needs: ["lint:persistence", "compile:persistence"]
  script:
    - cd stackgate_persistence
    - mkdir cover
    - mix test --trace --no-color | tee cover/test_report.txt
  dependencies:
    - compile:persistence
  artifacts:
    expire_in: 15 min
    paths:
       - stackgate_persistence/cover

test:engine:
  <<: *elixir_default
  stage: test:engine
  needs: ["lint:engine", "compile:engine"]
  script:
    - cd stackgate_engine
    - mkdir cover
    - mix coveralls.html --trace --no-color | tee cover/test_report.txt
  dependencies:
    - compile:engine
  artifacts:
    expire_in: 15 min
    paths:
       - stackgate_engine/cover


test:interface:
  <<: *elixir_default
  stage: test:interface
  needs: ["lint:interface", "test:persistence", "test:engine", "compile:interface"]
  script:
    - cd stackgate_interface
    - mkdir cover
    - mix test --trace --no-color | tee cover/test_report.txt
  dependencies:
    - compile:interface
  artifacts:
    expire_in: 15 min
    paths:
       - stackgate_interface/cover

ci/create.yml:

.elixir_default: &elixir_default
  image: wyrdforge/elixir:1.11.1-alpine
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

create:documentation:
  <<: *elixir_default
  stage: create:deployment
  needs: ["test:persistence", "test:engine", "test:interface", "compile:persistence", "compile:engine", "compile:interface"]
  script:
    - cd stackgate_persistence
    - mix docs --output ../stackgate_persistence/docs/
    - cd ../stackgate_engine
    - mix docs --output ../stackgate_engine/docs/
    - cd ../stackgate_interface
    - mix docs --output ../stackgate_interface/docs/
  dependencies:
    - compile:persistence
    - compile:engine
    - compile:interface
    - test:engine
    - test:persistence
    - test:interface
  artifacts:
    paths:
      - stackgate_engine/cover
      - stackgate_persistence/cover
      - stackgate_interface/cover
      - stackgate_engine/docs
      - stackgate_persistence/docs
      - stackgate_interface/docs

create:app_container:
  <<: *elixir_default
  stage: create:deployment
  needs: ["test:persistence", "test:engine", "test:interface"]
  variables:
    MIX_ENV: prod
  script:
    - echo "We will use a multistage Dockerfile for compiling a distillery release here and build the container for staging and production."
    - echo "Automatic deployment to staging and manual deployment after revision to prod will follow later on in the 'deploy' stage."
  dependencies: []
  

Attention: The job create:app_container must explicitly clear dependencies. This way, we omit all test/dev-debris in artifacts for this job and we can compile from scratch with MIX_ENV=prod.