Phoenix improve test coverage

Hi @i-n-g-m-a-r I’m confused you’re asking a question but then also linking to a blog post that answers your question. Can you elaborate as to what you’re looking for?

Most, if not all, of these files are pretty safe to put in a ignore clause for code coverage. I mean how would you meaningfully have a 100% test coverage of a telemetry module that’s mostly generated code?

Hi Ben,

I’m looking for actual test coverage.
With regard to the (ignore_modules) answer this would feel acceptable:

test_coverage: [
  ignore_modules: [
    # DemoWeb.Layouts,
    # DemoWeb.PageHTML,
    # DemoWeb.CoreComponents,
    Demo.DataCase,
    Demo.Repo,
    # DemoWeb.ErrorHTML,
    Demo.Application,
    DemoWeb.Router,
    DemoWeb.Telemetry
  ]
]

To exclude every single view does not feel acceptable.
I don’t know how to make a view pass any test.
With regard to CoreComponents it would be nice to have a working example.
Then we could update the tests after making changes to (modules using) CoreComponents.

While on the topic of testing phoenix applications,
I find myself putting this kind of not very pretty code into ConnCase:

using do
  quote do
    # ...

    def put_session(conn, session) when is_map(session) do
      Enum.reduce session, conn, fn {k, v}, c ->
        put_session c, k, v
      end
    end

    def run(conn, session \\ %{}, fun) do
        old = conn
              |> build_conn
              |> put_session(session)
              |> load_cookies

        ref = "#{conn.scheme}://#{conn.host}"
              |> Path.join(conn.request_path)

        build_conn()
        |> put_skb
        |> Plug.Test.recycle_cookies(old)
        |> put_req_header("referer", ref)
        |> fun.()
    end

    def build_conn(%{resp_cookies: resp_cookies}) do
      build_conn()
      |> put_skb
      |> put_cookies(resp_cookies)
      |> fetch_session
    end

    defp put_skb(%_{secret_key_base: nil} = conn) do
      conn
      |> Map.put(:secret_key_base, @skb)
      |> Plug.Session.call(@session_opts)
    end

    defp put_skb(conn), do: conn

    defp load_cookies(%_{private: private} = conn) do
      Enum.reduce(private[:before_send] || [], %{conn | state: :set}, & &1.(&2))
    end

    defp put_cookies(conn, resp_cookies \\ nil) do
      Enum.reduce resp_cookies || conn.resp_cookies, conn, fn {key, %{value: value}}, acc ->
        put_req_cookie acc, key, value
      end
    end
  end
end

In order to test all kinds of scenario’s involving sessions / cookies.
Probably this kind of code shouldn’t be necessary.

The generated code could be tested / covered ?

Why should it? And how do you test the generated telemetry module, for example? It serves as a callback / handler that’s registered elsewhere; not in your application logic.

What do you mean, what about 100% coverage ?

BTW totally agree, this is definitely not the responsibility of the developer to test generated modules, even if they are theoretically now part of your application.

Taking a bit of a step back, I wonder if there is perhaps gaps in how the test coverage is measured? If he’s got basic tests to render say the front page you’d expect that to run the DemoWeb.PageHTML and Layouts modules.

To be fair I don’t understand even to this day what this metric achieves, the fact that tests execute 100% of your code, doesn’t mean that the tests won’t miss a bug.

In the world of agile development, where we make software that evolves over time, having 100% coverage is like shooting yourself in the leg, a lot of effort to write tests, tests are brittle because they cover all implementation details, a lot of effort to change code and rewrite tests.

I even worked at a company where they implemented mutation testing, where it would be literally impossible to change a line of code without a couple of tests failing.

Hi Ben,

This is a regular pattern:

     0.00% | MyappWeb.ThingHTML
   100.00% | MyappWeb.ThingController

Where ThingController is tested using (also) lots of different ConnCase calls (get, post, etc).

Personally I would trust phoenix code to be well tested.
So most of the time I’m more rigorous regarding my own code.
Sometimes though I will collaborate with or work for people that demand good coverage.
Starting with near 100% would be nice.