Code coverage tools for Elixir?

Hey there, I just wrote this article:

Sure Look at my repo here:

https://github.com/cblavier/phx_component_helpers

Does codecov.io give you a coverage and status badge?

Just a brief update, because I needed to get solid code coverage for a decent phoenix app (umbrella app of 5 sub-apps, 17k loc).

On the app side, we’ve been using excoveralls : it is reliable, can be run in umbrella apps, with partitioned parallel testing and can output in different formats.

Then we needed also a good UI to be able to drill down within our application code to know what to test to improve our code coverage. I tested:

• coveralls.io: the UI is only OK but not really convenient nor pretty. I could not make Github Status checks or comments to work (in order to get instant code coverage feedback when opening a pull request). Integration in the CI was not really convenient as well because we had to disable partitioned testing to run all tests at once and upload a single report.

• codeclimate.com was disapointing Integration within CI was ok : we had to generate multiple json reports (one per partition) with excoveralls then format them, merge them and upload the whole with cc_test_reporter (binary provided by code_climate). Github feedback is working well, but you can’t really explore your code source tree within codeclimate UI. No go

• codecov.io is our choice. Integration within the CI is really simple : you can upload multiple report for a single build (one per test partition) and everything will be merged on the serverside. Github feedback is really insightful and the UI gives you the ability to drill down as you want to build your test coverage strategy

CleanShot 2021-02-23 at 11.23.04@2x1950×2358 217 KB

Codecov sunburst GIF here

I would be extremely hard to do because of two things:

1. the very tenuous mapping between Elixir’s AST and the macro-expanded version. This makes it very hard to count which branches we covered, because the branches in the macro-expanded code are not necessarily related to the branches in the original code

2. It’s very hard to get the Elixir AST for a given module. You’d have to use files as compilation units, and not modules.

Maybe it’s not as hard as I think it is, but it’s certainly not very easy. It’s much easir to get access to and manipulate the Erlang abstract format. I mean, I can’t even find a way to expand something inside a defmodule into Elixir AST. That’s probably because defmodule acts more like a function that creates a module and injects it into the running BEAM instance than a macro that is expanded into something more basic.

I guess this probably means there isn’t even a “fully expanded ast” for me to look at. But there must be something that the Elixir compiler feeds into the Erlang compiler, right?

I believe what @hauleth was driving at is that you can restructure that code as

def handle_has_key(true), do: b
def handle_has_key(false), do: c

def main do
  handle_has_key(Map.key?(a))
end

i.e. pass the result of the check into the function

Of course, I’m not advocating to write all your code in this manner, just pointing out that it is possible

I have badly misread while I was on the phone. Apologies.

No exactly, the difference is that JS is interpreted, so it has no notion of “compiled code”. In Erlang on the other hand there is such possibility, and single line can result in multiple VM “statements”, for example:

Integer.to_string(a + 1)

Will result with 2 instructions:

    {line,[{location,"lib/tasks.ex",6}]}.
    {gc_bif,'+',{f,0},1,[{x,0},{integer,1}],{x,0}}.
    {line,[{location,"lib/tasks.ex",6}]}.
    {call_ext_only,1,{extfunc,erlang,integer_to_binary,1}}.

So you can see that it is hard to match statements to instructions, especially as in theory compiler is free to reorder commands as it pleases as long as the result is the same (this can be especially seen in C/C++ where it is written in standard, and result of it is that foo(a++, a++) is undefined behaviour).

FWIW the current cover tool does not use the assembly stored in .beam files. What it does is to retrieve the Erlang AST for the module, transform it and recompile.

The transformation consists primarily of introducing calls to :ets.update_counter/3 each time the line annotation of the AST changes. The analysis is just counting how many lines could be called vs how many were actually called.

An Elixir version that worked on Elixir AST in a similar way wouldn’t be extremely hard to do. It could also inject the counter calls for each branch, etc, to achieve the desired coverage.

This is interesting. It appears that even though no .beam file is created, you are still correct !

I would agree with you if only guards weren’t limited. The limitations with guards directly imply that some conditions cannot be converted into the pattern matching formula. For example, Map.key? can’t be used in a guard. if you have

if Map.key?(a), do: b, else: c

Then you are stuck.

Interesting discussion overall, can’t wait to see where it leads!

Conclusions so far:

1. Differentiating between lines and statements makes no sense in Erlang/Elixir because BEAM doesn’t have the concept of statements. For BEAM, every line is a statement.