I use this custom formatter for our CI to get the failures summarized at the bottom. There is a small race condition in it which can cause an extra line or two at the end, but it does the job. I’ve only ever used with along with --trace so not sure what it looks like with anything else (probably much the same).

defmodule MyApp.Test.Formatter do
  @moduledoc """
  Adds test failures to the bottom of the test output.

  This is mostly only useful when run with `--trace`.

  Usage:

     $ mix test --trace --formatter MyApp.Test.Formatter
  """

  use GenServer

  @format_cols 80

  @doc false
  def init(opts) do
    {:ok, pid} = GenServer.start_link(ExUnit.CLIFormatter, opts)
    {:ok, %{cli_formatter: pid, failures: []}}
  end

  def handle_cast({:test_finished, %{state: {:failed, errors}} = test} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    counter = length(state.failures) + 1

    error =
      ExUnit.Formatter.format_test_failure(test, errors, counter, @format_cols, &formatter/2)

    state = %{state | failures: [error | state.failures]}

    {:noreply, state}
  end

  def handle_cast({:suite_finished, _} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    if Enum.any?(state.failures) do
      failures =
        state.failures
        |> Enum.reverse()
        |> Enum.join("\n")

      IO.puts("""

      #{String.duplicate("=", @format_cols)}

        Failures:

      #{failures}

      #{String.duplicate("=", @format_cols)}
      """)
    end

    {:noreply, state}
  end

  def handle_cast(event, state) do
    GenServer.cast(state.cli_formatter, event)

    {:noreply, state}
  end

  defp formatter(_key, value), do: value
end

I can’t answer your question directly – it’s been a long time since I last looked at test formatters – but one workaround is to just run mix test --failed afterwards, which will only run the tests that failed during the previous run.

If your concern is that you have too much test terminal output and that you’re finding it hard to track the output only of those that failed then maybe that command will help you.

Yes, that’s exactly my concern. Thanks for pointing me to mix test --failed. Unfortunately, it doesn’t help in case when failed test log is captured so you still have a lot of scrolling. Sometimes you just want to see the summary of failed tests - without investigating the log of the failed test.