I spent a bit of time looking at getting code-coverage metrics and reporting working for an umbrella project in a self-hosted GitLab instance (15.3). There were a few things I tried along the way and I’m posting it here in the hope that it might save others a little bit of time in the future. Also I don’t have a blog, and no interest in maintaining one, so this will be my own aide memoir.
Regarding GitLab, there appear to be basically two different aspects of support for code-coverage. First, a single metric providing a % measure of total code-coverage which is used in a number of places:
- On pipelines, i.e. what is the coverage of a particular build
- On merge requests, which also tells us the delta vs the current coverage of the default branch, typically
main
- As a charted time-series value in ‘Analytics’
- As a value you can stick on a badge, also taken from the default branch
The second, is a more detailed view, allowing you to visualise the actual changes to coverage in individual files within an existing merge request (discussed in more detail here).
With that out of the way, here’s what I landed on:
Run all the tests with coverage
In the .gitlab-ci.yml
:
script:
- MIX_ENV=test mix test --cover export-coverage default
Note, I didn’t need to add any test_coverage:
configuration to any of the mix projects, nor the root one.
Calculate the total test coverage
This is what will be used for pipelines, stats and badges.
after_script:
# Generate test coverage metric
- MIX_ENV=test mix test.coverage || true
...
coverage:
- '/ Coverage: \d+.\d+%/'
Generate Cobertura files
In order to present coverage in merge requests, GitLab requires that coverage reports in Cobertura format be available. Generating a single coverage file from all umbrella applications was achieved using covertool.
In the root mix.exs
, add the following dependency:
def deps do
[
{:covertool, "~> 2.0.4", only: :test, runtime: false, app: false, compile: "rebar3 escriptize"}
]
end
Then, in the after_script:
section of the .gitlab-ci.yml
, have covertool build a single coverage.xml
file from the raw coverage data generated during the test run, first by using cover to combine the individual coverage files into one, and then using covertool to convert that into cobertura format. In the example below, there are 3 projects in the umbrella, imaginatively named :proj_a
, :proj_b
and :proj_c
.
after_script:
# Generate test coverage metric
- MIX_ENV=test mix test.coverage || true
# Build a single coverage file, all.coverdata
- elixir
-e "Enum.each(~w(proj_a proj_b proj_c), fn app -> :cover.import(~c(./apps/#{app}/cover/default.coverdata)) end)"
-e ":cover.export(~c(./all.coverdata))"
# Convert it to cobertura format in a single coverage.xml file
- ./deps/covertool/_build/default/bin/covertool
-cover all.coverdata
-output coverage.xml
-ebin _build/test/lib/proj_a/ebin,_build/test/lib/proj_b/ebin,_build/test/lib/proj_c/ebin
# Fix up the paths in the resulting file
- sed -i "/\/builds\/umbrella\///g" coverage.xml
The final step fixes up the paths in the coverage.xml
file so the <filename>
elements contain the same path that GitLab uses in the merge requests. In the above example this has the affect of changing
<filename>/builds/umbrella/apps/...</filename
to <filename>apps/...</filename
.
Upload the coverage report
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
A final note - in the above examples I’ve used multiline yaml in some of the commands to improve the formatting, but in my own system it’s not been written and tested that way so you might need to tweak it a bit.