Phoenix improve test coverage

How to improve phoenix test coverage?

Source

$ mix test --cover
......
Generating cover results ...

Percentage | Module
-----------|--------------------------
     0.00% | DemoWeb.Layouts
     0.00% | DemoWeb.PageHTML
    16.67% | DemoWeb.CoreComponents
    25.00% | Demo.DataCase
    50.00% | Demo.Repo
    50.00% | DemoWeb.ErrorHTML
    75.00% | Demo.Application
    75.00% | DemoWeb.Router
    80.00% | DemoWeb.Telemetry
   100.00% | Demo
   100.00% | Demo.Mailer
   100.00% | DemoWeb
   100.00% | DemoWeb.ConnCase
   100.00% | DemoWeb.Endpoint
   100.00% | DemoWeb.ErrorJSON
   100.00% | DemoWeb.Gettext
   100.00% | DemoWeb.PageController
-----------|--------------------------
    27.41% | Total

Coverage test failed, threshold not met:

    Coverage:   27.41%
    Threshold:  90.00%

Generated HTML coverage results in "cover" directory

Hi @i-n-g-m-a-r I’m confused you’re asking a question but then also linking to a blog post that answers your question. Can you elaborate as to what you’re looking for?

Most, if not all, of these files are pretty safe to put in a ignore clause for code coverage. I mean how would you meaningfully have a 100% test coverage of a telemetry module that’s mostly generated code?

Hi Ben,

I’m looking for actual test coverage.
With regard to the (ignore_modules) answer this would feel acceptable:

test_coverage: [
  ignore_modules: [
    # DemoWeb.Layouts,
    # DemoWeb.PageHTML,
    # DemoWeb.CoreComponents,
    Demo.DataCase,
    Demo.Repo,
    # DemoWeb.ErrorHTML,
    Demo.Application,
    DemoWeb.Router,
    DemoWeb.Telemetry
  ]
]

To exclude every single view does not feel acceptable.
I don’t know how to make a view pass any test.
With regard to CoreComponents it would be nice to have a working example.
Then we could update the tests after making changes to (modules using) CoreComponents.

While on the topic of testing phoenix applications,
I find myself putting this kind of not very pretty code into ConnCase:

using do
  quote do
    # ...

    def put_session(conn, session) when is_map(session) do
      Enum.reduce session, conn, fn {k, v}, c ->
        put_session c, k, v
      end
    end

    def run(conn, session \\ %{}, fun) do
        old = conn
              |> build_conn
              |> put_session(session)
              |> load_cookies

        ref = "#{conn.scheme}://#{conn.host}"
              |> Path.join(conn.request_path)

        build_conn()
        |> put_skb
        |> Plug.Test.recycle_cookies(old)
        |> put_req_header("referer", ref)
        |> fun.()
    end

    def build_conn(%{resp_cookies: resp_cookies}) do
      build_conn()
      |> put_skb
      |> put_cookies(resp_cookies)
      |> fetch_session
    end

    defp put_skb(%_{secret_key_base: nil} = conn) do
      conn
      |> Map.put(:secret_key_base, @skb)
      |> Plug.Session.call(@session_opts)
    end

    defp put_skb(conn), do: conn

    defp load_cookies(%_{private: private} = conn) do
      Enum.reduce(private[:before_send] || [], %{conn | state: :set}, & &1.(&2))
    end

    defp put_cookies(conn, resp_cookies \\ nil) do
      Enum.reduce resp_cookies || conn.resp_cookies, conn, fn {key, %{value: value}}, acc ->
        put_req_cookie acc, key, value
      end
    end
  end
end

In order to test all kinds of scenario’s involving sessions / cookies.
Probably this kind of code shouldn’t be necessary.

The generated code could be tested / covered ?

Why should it? And how do you test the generated telemetry module, for example? It serves as a callback / handler that’s registered elsewhere; not in your application logic.

What do you mean, what about 100% coverage :joy: ?

BTW totally agree, this is definitely not the responsibility of the developer to test generated modules, even if they are theoretically now part of your application.

1 Like

Taking a bit of a step back, I wonder if there is perhaps gaps in how the test coverage is measured? If he’s got basic tests to render say the front page you’d expect that to run the DemoWeb.PageHTML and Layouts modules.

1 Like

To be fair I don’t understand even to this day what this metric achieves, the fact that tests execute 100% of your code, doesn’t mean that the tests won’t miss a bug.

In the world of agile development, where we make software that evolves over time, having 100% coverage is like shooting yourself in the leg, a lot of effort to write tests, tests are brittle because they cover all implementation details, a lot of effort to change code and rewrite tests.

I even worked at a company where they implemented mutation testing, where it would be literally impossible to change a line of code without a couple of tests failing.

1 Like

Hi Ben,

This is a regular pattern:

     0.00% | MyappWeb.ThingHTML
   100.00% | MyappWeb.ThingController

Where ThingController is tested using (also) lots of different ConnCase calls (get, post, etc).

Personally I would trust phoenix code to be well tested.
So most of the time I’m more rigorous regarding my own code.
Sometimes though I will collaborate with or work for people that demand good coverage.
Starting with near 100% would be nice.

That’s fair, but I think that’s a separate discussion. The OP has a specific technical question around how code coverage works with these modules, let’s leave debate on code coverage as a metric for another thread.

1 Like

Well, that could be interesting to find out, i.e. how to make sure generated modules are tested more comprehensively.

@i-n-g-m-a-r Maybe dumping the generated code – after macro expansion – in a separate file and inspecting it is a good first step.

Thanks for your suggestion Dimitar.
In my cases I will think “it’s me” like in pebcac.
Most of the time I’m right over the target.
With regard to testing and macro’s I don’t seem to be the problem.

I copied all of the files here:

into my test directory.
It doesn’t do anything to improve coverage.

Check this out.

  test "view coverage" do
    views = [
      MyappWeb.Layouts,
      MyappWeb.ThingHTML,
      MyappWeb.StuffHTML,
      MyappWeb.MoreStuffHTML,
      MyappWeb.PageHTML
    ]
    Enum.each views, fn view ->
      assert :ok == apply(view, :__phoenix_verify_routes__, [nil])
    end
  end
   100.00% | MyappWeb.Layouts
   100.00% | MyappWeb.ThingHTML
   100.00% | MyappWeb.StuffHTML
   100.00% | MyappWeb.MoreStuffHTML
   100.00% | MyappWeb.PageHTML

As @benwilson512 said, it’s odd that in a vanilla project that Layouts and PageHTML are not being covered as the generated PageControllerTest is definitely exercising these.

In terms of CoreComponents, you will start to see coverage through your controller and live views test as you start using them. Of course, just adding a <.button>hello</.button> to PageControllerTest will increase the coverage without asserting anything about it, so I take coverage with a grain of salt.

1 Like