Code Coverage incorrect for function with multiple clauses

Hi all!

I noticed something odd while testing one of my LiveViews today. I wrote a test for one of the handle_event/3 function clauses. Made the test pass. Verified that the assigns were getting set correctly. Then discovered that code coverage wasn’t marking that code as having been run. I added an IO.inspect/2, saw the log in my test output, but code coverage still says that IO.inspect line was never run.

I was able to make a very minimal reproduction in a brand new mix new project:

elixir 1.17.3-otp-27
erlang 27.1

defmodule CoverageBug do
  def hello do
    check("target", nil)
    :world
  end

  def check("one", _) do
    nil
  end

  def check("two", _) do
    nil
  end

  def check(_, %{some_match: :three}) do
    nil
  end

  def check("four", _) do
    nil
  end

  def check("target", _) do
    IO.inspect("This line is run but not covered")
    nil
  end
end

defmodule CoverageBugTest do
  use ExUnit.Case

  test "greets the world" do
    assert CoverageBug.hello() == :world
  end
end

When running mix test --cover, you’ll see the IO.inspect line logged out in the test output. But it won’t show up as covered in the test coverage report. If you delete any of those first 4 check function clauses, then the IO.inspect in the 5th clause will start being reported as covered. Has anyone experienced anything like this? Am I missing something obvious?

I ended up trying this with a different version of erlang and it worked:

elixir 1.17.3-otp-26
erlang 26.2.5.3

So looks like this was a regression in the latest erlang major release.

1 Like

Thanks for sharing. I don’t personally look at these numbers, but I know we have CI tools at work that do – and they output curious results, so this might be related.

Appriciate your signal.