Code Coverage for Umbrella Projects in GitLab

I spent a bit of time looking at getting code-coverage metrics and reporting working for an umbrella project in a self-hosted GitLab instance (15.3). There were a few things I tried along the way and I’m posting it here in the hope that it might save others a little bit of time in the future. Also I don’t have a blog, and no interest in maintaining one, so this will be my own aide memoir.

Regarding GitLab, there appear to be basically two different aspects of support for code-coverage. First, a single metric providing a % measure of total code-coverage which is used in a number of places:

• On pipelines, i.e. what is the coverage of a particular build
• On merge requests, which also tells us the delta vs the current coverage of the default branch, typically main
• As a charted time-series value in ‘Analytics’
• As a value you can stick on a badge, also taken from the default branch

The second, is a more detailed view, allowing you to visualise the actual changes to coverage in individual files within an existing merge request (discussed in more detail here).

With that out of the way, here’s what I landed on:

Run all the tests with coverage

In the .gitlab-ci.yml:

script:
  - MIX_ENV=test mix test --cover export-coverage default

Note, I didn’t need to add any test_coverage: configuration to any of the mix projects, nor the root one.

Calculate the total test coverage

This is what will be used for pipelines, stats and badges.

after_script:
  # Generate test coverage metric
  - MIX_ENV=test mix test.coverage || true
...
coverage:
  - '/    Coverage:   \d+.\d+%/'

Generate Cobertura files

In order to present coverage in merge requests, GitLab requires that coverage reports in Cobertura format be available. Generating a single coverage file from all umbrella applications was achieved using covertool.

In the root mix.exs, add the following dependency:

def deps do
  [
    {:covertool, "~> 2.0.4", only: :test, runtime: false, app: false, compile: "rebar3 escriptize"}
  ]
end

Then, in the after_script: section of the .gitlab-ci.yml, have covertool build a single coverage.xml file from the raw coverage data generated during the test run, first by using cover to combine the individual coverage files into one, and then using covertool to convert that into cobertura format. In the example below, there are 3 projects in the umbrella, imaginatively named :proj_a, :proj_b and :proj_c.

after_script:
  # Generate test coverage metric
  - MIX_ENV=test mix test.coverage || true
  # Build a single coverage file, all.coverdata
  - elixir 
    -e "Enum.each(~w(proj_a proj_b proj_c), fn app -> :cover.import(~c(./apps/#{app}/cover/default.coverdata)) end)"
    -e ":cover.export(~c(./all.coverdata))"
  # Convert it to cobertura format in a single coverage.xml file
  - ./deps/covertool/_build/default/bin/covertool
    -cover all.coverdata
    -output coverage.xml
    -ebin _build/test/lib/proj_a/ebin,_build/test/lib/proj_b/ebin,_build/test/lib/proj_c/ebin
  # Fix up the paths in the resulting file
  - sed -i "/\/builds\/umbrella\///g" coverage.xml

The final step fixes up the paths in the coverage.xml file so the <filename> elements contain the same path that GitLab uses in the merge requests. In the above example this has the affect of changing
<filename>/builds/umbrella/apps/...</filename to <filename>apps/...</filename.

Upload the coverage report

artifacts:
  reports:
    coverage_report:
      coverage_format: cobertura
      path: coverage.xml

A final note - in the above examples I’ve used multiline yaml in some of the commands to improve the formatting, but in my own system it’s not been written and tested that way so you might need to tweak it a bit.