Failed tests summary at the end of mix test output

Hi,

I have one basic question.

Is there a way to get the list of failed tests at the end of mix test output? Do you have to come up with custom formatter or there is some flag?

Cheers

1 Like

I can’t answer your question directly – it’s been a long time since I last looked at test formatters – but one workaround is to just run mix test --failed afterwards, which will only run the tests that failed during the previous run.

If your concern is that you have too much test terminal output and that you’re finding it hard to track the output only of those that failed then maybe that command will help you.

2 Likes

Yes, that’s exactly my concern. Thanks for pointing me to mix test --failed. Unfortunately, it doesn’t help in case when failed test log is captured so you still have a lot of scrolling. Sometimes you just want to see the summary of failed tests - without investigating the log of the failed test.

1 Like

It would be great to be able to pass a function to ExUnit.start/1 in order to run some custom code after the tests are finished.

For now you can use the following code in an exs file:

manifest = "_build/test/lib/pleenk_bb/.mix/.mix_test_failures"

if not File.exists?(manifest) do
  System.stop()
  Process.sleep(:infinity)
end

manifest
|> File.read!()
|> :erlang.binary_to_term()
|> elem(1)
|> Enum.group_by(fn {_, file} -> file end, fn {{_mod, name}, _file} -> name end)
|> Enum.each(fn {file, tests} ->
  IO.puts([
    IO.ANSI.red(),
    "Failures in #{file}:\n\n",
    Enum.map_join(tests, "\n", &"* #{&1}"),
    "\n",
    IO.ANSI.reset()
  ])
end)

I use this custom formatter for our CI to get the failures summarized at the bottom. There is a small race condition in it which can cause an extra line or two at the end, but it does the job. I’ve only ever used with along with --trace so not sure what it looks like with anything else (probably much the same).

defmodule MyApp.Test.Formatter do
  @moduledoc """
  Adds test failures to the bottom of the test output.

  This is mostly only useful when run with `--trace`.

  Usage:

     $ mix test --trace --formatter MyApp.Test.Formatter
  """

  use GenServer

  @format_cols 80

  @doc false
  def init(opts) do
    {:ok, pid} = GenServer.start_link(ExUnit.CLIFormatter, opts)
    {:ok, %{cli_formatter: pid, failures: []}}
  end

  def handle_cast({:test_finished, %{state: {:failed, errors}} = test} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    counter = length(state.failures) + 1

    error =
      ExUnit.Formatter.format_test_failure(test, errors, counter, @format_cols, &formatter/2)

    state = %{state | failures: [error | state.failures]}

    {:noreply, state}
  end

  def handle_cast({:suite_finished, _} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    if Enum.any?(state.failures) do
      failures =
        state.failures
        |> Enum.reverse()
        |> Enum.join("\n")

      IO.puts("""

      #{String.duplicate("=", @format_cols)}

        Failures:

      #{failures}

      #{String.duplicate("=", @format_cols)}
      """)
    end

    {:noreply, state}
  end

  def handle_cast(event, state) do
    GenServer.cast(state.cli_formatter, event)

    {:noreply, state}
  end

  defp formatter(_key, value), do: value
end
5 Likes

FWIW, in case this is useful for anyone in the future, here is our test: target in our Makefile:

$ cat Makefile
...
test:
         ...
         time mix test --cover --trace --slowest 4 --timeout 20 --warnings-as-errors || \
                 (mix test --failed; exit 100)
...

If mix test fails, it runs mix test --failed, and returns a non-zero code (in case the failing tests were flakey or the failure was caused by --cover or --warnings-as-errors:

We run make test on our dev machines and in our CI/CD pipelines, and this has been a time saver for us.