Surprising lines missing from coverage results

Please open up a bug report, we can likely fix the tuple and list ones, but not the atom because literals do not have line numbers in the AST.

EDIT: I have fixed this locally and I will push it soon.

Amazed with your response!

https://github.com/elixir-lang/elixir/issues/10847

I think I just ran into the same issue.

Lines in the source files containing simple terms, that have no metadata in thier AST representations (e.g.: single atoms, 2-tuples with only no-metadata elements, lists os any length with only no-metadata elements, binary literals, things like that) are kicked out of code coverage tests, which is a big problem. (For instance, a function or case clauses returning :ok, {:error, :not_found} and such simple expressions are just not counted.) I can wrap them into elem({X}, 0) where X is the simple expression to force the coverage tests to work properly, but this is just plain ugly.

Is it an option to somehow change the Elixir compiler in order to compile the source with some tricks like the above one, when compiling for the :test env? This does not change the semantics of the code, but fixes the coverage problem.

What is your Elixir version? We have improved this in more recent releases.

I tried it with Elixir 1.12.2 and 1.12.3 (on Erlang/OTP 23.2.3). Also tried it with 1.12.3 on Erlang/OTP 24.0.6.
:ok and {:error, :reason} lines are not included in coverage reports on these versions for me.

Yup, I could reproduce it. Fixes coming to Elixir master shortly.

Found the commit on master, thanks for the improvements. As far as I can see, case patterns are now covered, which is great. It is now visible which case branch was tested. But bodies with atoms and 2-tuples are still missing. I also miss the else line of an if macro. Here only the if ... do part comes up in the coverage test report. The :error and the else lines don’t:

    if ... do # covered line
      :ok
    else
      :error
    end

In case of a case, the case ... do line and the patterns are included now, not the atom and 2-tuple:

    case ... do   # covered
      pattern1 -> # covered
        :ok       # MISSING !!
      pattern2 -> # covered
        {:error, :not_ok} # MISSING !!
    end

Is it impossible to also cover those terms with no metadata in their AST? (Actually, if you could cover the clause bodies, then case pattern coverage has little information.)

They are not missing per se. If the pattern was covered, then by definition what immediately follows the pattern is being covered too. We can’t highlight it because we don’t have precise line information, but the coverage of the pattern is enough in those cases.

Well, in case of the case statement, it is fine. But the if statement above is a good example when this is not enough. The if .. do line is highlighted, but none of the following 4 lines are, so we cannot know if we covered both the then branch and the else branch, or just one of them.

Isn’t it a solution if you wrap those values without line information into an identity function call when compiling for test env? If you could compile them into something with line info in it, that, when executed, produces the original value, that would be great.

Thank for fixing the case problem, José. I can say that almost everything is fine with coverage tests now. There are only a few parts missing, namely the if, unless and the with special forms still need some fixing. I can give you example codes about the problem:

with

  with {:ok, file_content} <- File.read("/tmp/xxx"),
       {:ok, _value} <- parse(file_content)
  do
    :successfully_parsed
  else
    {:error, :noent} ->
      :file_not_found

    :error ->
      :content_not_parseabel

    _ ->
      :other_error
  end

The problem with this code is that none of the lines between the do and the end lines are flagged for coverage. Only the first two lines of the code above can be green or red. (I put the do to a new line just to see if it gets red/green, but unfortunately it didn’t.)

The good news is that try has a very similar else branch, that works perfectly well, so maybe this can be fixed just by mimicking how try..rescue..else works.

if and unless

1  if x == 42 do
2    "42!!!"
3  else
4    :other_value
5  end

In this case lines from 2 to 5 are not coverable (not green, nor red, just plain white). The exact same is true is you replace if with unless. This is very sad.

I also have to mention that there are other special forms that are fine. case and cond are now OK, just like receive (even with after), try and def function definitions.

Is there any hope these errors are gonna be fixed?