Surprising lines missing from coverage results

Hey Elixir-land!

It’s been a while since I’ve been around. Getting back into lots of elixir work and I found something unexpected regarding the coverage system. Essentially function calls wrapped in literals don’t seem to get counted as coverable statements.

Steps to reproduce:

  1. run mix new coverage_why
  2. paste in lib/coverage_why.ex
defmodule CoverageWhy do
  def hello do
    :world
  end

  def i_has_coverage do
    hello()
  end

  def i_has_no_coverage do
    {:ok, hello()}
  end

  def i_has_no_coverage_as_well do
    [hello()]
  end

  def i_has_no_coverage_as_either do
    [hello()]

    :ok
  end
end
  1. run mix test --cover

I get the results of

Expectations

I would expect all the lines except the original def hello... to be uncovered. If I had to guess the wrapping the call in a literal causes the function call to be inlined somehow. Is this the expected coverage results and is it documented anywhere?

3 Likes

Please open up a bug report, we can likely fix the tuple and list ones, but not the atom because literals do not have line numbers in the AST.

EDIT: I have fixed this locally and I will push it soon.

15 Likes

Amazed with your response!

1 Like

I think I just ran into the same issue.

Lines in the source files containing simple terms, that have no metadata in thier AST representations (e.g.: single atoms, 2-tuples with only no-metadata elements, lists os any length with only no-metadata elements, binary literals, things like that) are kicked out of code coverage tests, which is a big problem. (For instance, a function or case clauses returning :ok, {:error, :not_found} and such simple expressions are just not counted.) I can wrap them into elem({X}, 0) where X is the simple expression to force the coverage tests to work properly, but this is just plain ugly.

Is it an option to somehow change the Elixir compiler in order to compile the source with some tricks like the above one, when compiling for the :test env? This does not change the semantics of the code, but fixes the coverage problem.

What is your Elixir version? We have improved this in more recent releases.

1 Like

I tried it with Elixir 1.12.2 and 1.12.3 (on Erlang/OTP 23.2.3). Also tried it with 1.12.3 on Erlang/OTP 24.0.6.
:ok and {:error, :reason} lines are not included in coverage reports on these versions for me. :frowning:

1 Like

Yup, I could reproduce it. Fixes coming to Elixir master shortly.

9 Likes

Found the commit on master, thanks for the improvements. As far as I can see, case patterns are now covered, which is great. It is now visible which case branch was tested. But bodies with atoms and 2-tuples are still missing. I also miss the else line of an if macro. Here only the if ... do part comes up in the coverage test report. The :error and the else lines don’t:

    if ... do # covered line
      :ok
    else
      :error
    end

In case of a case, the case ... do line and the patterns are included now, not the atom and 2-tuple:

    case ... do   # covered
      pattern1 -> # covered
        :ok       # MISSING !!
      pattern2 -> # covered
        {:error, :not_ok} # MISSING !!
    end

Is it impossible to also cover those terms with no metadata in their AST? (Actually, if you could cover the clause bodies, then case pattern coverage has little information.)

They are not missing per se. If the pattern was covered, then by definition what immediately follows the pattern is being covered too. We can’t highlight it because we don’t have precise line information, but the coverage of the pattern is enough in those cases.

5 Likes

Well, in case of the case statement, it is fine. But the if statement above is a good example when this is not enough. The if .. do line is highlighted, but none of the following 4 lines are, so we cannot know if we covered both the then branch and the else branch, or just one of them.

Isn’t it a solution if you wrap those values without line information into an identity function call when compiling for test env? If you could compile them into something with line info in it, that, when executed, produces the original value, that would be great.