Surprising lines missing from coverage results

Hey Elixir-land!

It’s been a while since I’ve been around. Getting back into lots of elixir work and I found something unexpected regarding the coverage system. Essentially function calls wrapped in literals don’t seem to get counted as coverable statements.

Steps to reproduce:

  1. run mix new coverage_why
  2. paste in lib/coverage_why.ex
defmodule CoverageWhy do
  def hello do
    :world
  end

  def i_has_coverage do
    hello()
  end

  def i_has_no_coverage do
    {:ok, hello()}
  end

  def i_has_no_coverage_as_well do
    [hello()]
  end

  def i_has_no_coverage_as_either do
    [hello()]

    :ok
  end
end
  1. run mix test --cover

I get the results of

Expectations

I would expect all the lines except the original def hello... to be uncovered. If I had to guess the wrapping the call in a literal causes the function call to be inlined somehow. Is this the expected coverage results and is it documented anywhere?

3 Likes

Please open up a bug report, we can likely fix the tuple and list ones, but not the atom because literals do not have line numbers in the AST.

EDIT: I have fixed this locally and I will push it soon.

15 Likes

Amazed with your response!

1 Like

I think I just ran into the same issue.

Lines in the source files containing simple terms, that have no metadata in thier AST representations (e.g.: single atoms, 2-tuples with only no-metadata elements, lists os any length with only no-metadata elements, binary literals, things like that) are kicked out of code coverage tests, which is a big problem. (For instance, a function or case clauses returning :ok, {:error, :not_found} and such simple expressions are just not counted.) I can wrap them into elem({X}, 0) where X is the simple expression to force the coverage tests to work properly, but this is just plain ugly.

Is it an option to somehow change the Elixir compiler in order to compile the source with some tricks like the above one, when compiling for the :test env? This does not change the semantics of the code, but fixes the coverage problem.

What is your Elixir version? We have improved this in more recent releases.

1 Like

I tried it with Elixir 1.12.2 and 1.12.3 (on Erlang/OTP 23.2.3). Also tried it with 1.12.3 on Erlang/OTP 24.0.6.
:ok and {:error, :reason} lines are not included in coverage reports on these versions for me. :frowning:

1 Like

Yup, I could reproduce it. Fixes coming to Elixir master shortly.

9 Likes

Found the commit on master, thanks for the improvements. As far as I can see, case patterns are now covered, which is great. It is now visible which case branch was tested. But bodies with atoms and 2-tuples are still missing. I also miss the else line of an if macro. Here only the if ... do part comes up in the coverage test report. The :error and the else lines don’t:

    if ... do # covered line
      :ok
    else
      :error
    end

In case of a case, the case ... do line and the patterns are included now, not the atom and 2-tuple:

    case ... do   # covered
      pattern1 -> # covered
        :ok       # MISSING !!
      pattern2 -> # covered
        {:error, :not_ok} # MISSING !!
    end

Is it impossible to also cover those terms with no metadata in their AST? (Actually, if you could cover the clause bodies, then case pattern coverage has little information.)

They are not missing per se. If the pattern was covered, then by definition what immediately follows the pattern is being covered too. We can’t highlight it because we don’t have precise line information, but the coverage of the pattern is enough in those cases.

5 Likes

Well, in case of the case statement, it is fine. But the if statement above is a good example when this is not enough. The if .. do line is highlighted, but none of the following 4 lines are, so we cannot know if we covered both the then branch and the else branch, or just one of them.

Isn’t it a solution if you wrap those values without line information into an identity function call when compiling for test env? If you could compile them into something with line info in it, that, when executed, produces the original value, that would be great.

1 Like

Thank for fixing the case problem, José. I can say that almost everything is fine with coverage tests now. There are only a few parts missing, namely the if, unless and the with special forms still need some fixing. I can give you example codes about the problem:

with

  with {:ok, file_content} <- File.read("/tmp/xxx"),
       {:ok, _value} <- parse(file_content)
  do
    :successfully_parsed
  else
    {:error, :noent} ->
      :file_not_found

    :error ->
      :content_not_parseabel

    _ ->
      :other_error
  end

The problem with this code is that none of the lines between the do and the end lines are flagged for coverage. Only the first two lines of the code above can be green or red. (I put the do to a new line just to see if it gets red/green, but unfortunately it didn’t.)

The good news is that try has a very similar else branch, that works perfectly well, so maybe this can be fixed just by mimicking how tryrescueelse works.

if and unless

1  if x == 42 do
2    "42!!!"
3  else
4    :other_value
5  end

In this case lines from 2 to 5 are not coverable (not green, nor red, just plain white). The exact same is true is you replace if with unless. This is very sad.

I also have to mention that there are other special forms that are fine. case and cond are now OK, just like receive (even with after), try and def function definitions.

Is there any hope these errors are gonna be fixed?

Does this behavior persist if the lines contain something more complicated than a literal?

IIRC there’s a gotcha with coverage related to lines with a bare literal. Some related discussion: Save my from myself: Code coverage misses lines that return a literal

1 Like

Of course, not. This only happens when the AST representation of the line is not the usual 3-tuple with the metadata in the middle of it. (That means, if the representation is the term itself, like numbers, atoms, lists, 2-tuples, strings, maybe others.) The problem is that it is totally realistic that one writes a single atom, or a tuple like {:error, :not_found} into a clause body. This thing should work even with very simple literals. try works, so I believe it is possible to fix with too.

The fix is based on ->, so it can’t work on if/unless. with has issues because we mark some clauses as generated to make Dialyzer happy but this will be fixed when we depend on Erlang/OTP 25. Perhaps we can do conditional compilation to have the benefits sooner.

1 Like

Hi,

unfortunately this problem still exists. I totally understand the root cause of the issue (that is the missing line information in the quoted expressions), but found out that the else branch of a with special form actually has the line info in it, but coverage tools are not using it. Here is an oversimplified example (I use integers in it for two reasons: 1) ensure that no accidental line info is added, 2) make it easier to compare the Elixir code with the AST):

66       with 1 <- 2,
67           3 <- 4 do
68         5
69       else
70         6 ->
71           7
72
73         {8, 9, 10, 11, 12} ->
74           13
75
76        14 ->  
77           15 + 0
78   
79         _ ->
80           42
81       end

Here’s how it look like when code coverage is checked:

Screenshot from 2023-11-15 10-29-56

The <- lines are perfect. I understand why 5 (line 68) is not involved, that’s OK. But the else section looks strange. Only 15 + 0 is covered because of the + function call, not even the 5-tuple before the arrow.

Here’s the relevant part of the AST:

    else: [
      {:->, [line: 71, column: 9], [[6], 7]},
      {:->, [line: 74, column: 26],
       [[{:{}, [line: 74, column: 7], [8, 9, 10, 11, 12]}], 13]},
      {:->, [line: 77, column: 10],
       [[14], {:+, [line: 78, column: 12], [15, 0]}]},
      {:->, [line: 80, column: 9], [[{:_, [line: 80, column: 7], nil}], 42]}
    ]
  • As you can see all :-> 3-tuples have proper line metadata, but not used in coverage for some reason.
  • The ast of {8, 9, 10, 11, 12} has line metadata on the left hand side of the arrow, still not used in coverage.
  • Only when the right hand side of the arrow contains line metadata, then is the line used in coverage reports.

Is it possible to fix this somehow?

As mentioned earlier:

There is no option besides making with raise false warnings for those using Dialyzer :frowning: I thought we would be able to use Erlang’s maybe, but guard support is missing.