Failed tests summary at the end of mix test output

Questions / Help
1

Hi,

I have one basic question.

Is there a way to get the list of failed tests at the end of mix test output? Do you have to come up with custom formatter or there is some flag?

Cheers

1 Like
2

I can’t answer your question directly – it’s been a long time since I last looked at test formatters – but one workaround is to just run mix test --failed afterwards, which will only run the tests that failed during the previous run.

If your concern is that you have too much test terminal output and that you’re finding it hard to track the output only of those that failed then maybe that command will help you.

3 Likes
3

Yes, that’s exactly my concern. Thanks for pointing me to mix test --failed. Unfortunately, it doesn’t help in case when failed test log is captured so you still have a lot of scrolling. Sometimes you just want to see the summary of failed tests - without investigating the log of the failed test.

1 Like
4

It would be great to be able to pass a function to ExUnit.start/1 in order to run some custom code after the tests are finished.

For now you can use the following code in an exs file:

manifest = "_build/test/lib/pleenk_bb/.mix/.mix_test_failures"

if not File.exists?(manifest) do
  System.stop()
  Process.sleep(:infinity)
end

manifest
|> File.read!()
|> :erlang.binary_to_term()
|> elem(1)
|> Enum.group_by(fn {_, file} -> file end, fn {{_mod, name}, _file} -> name end)
|> Enum.each(fn {file, tests} ->
  IO.puts([
    IO.ANSI.red(),
    "Failures in #{file}:\n\n",
    Enum.map_join(tests, "\n", &"* #{&1}"),
    "\n",
    IO.ANSI.reset()
  ])
end)
1 Like
5

I use this custom formatter for our CI to get the failures summarized at the bottom. There is a small race condition in it which can cause an extra line or two at the end, but it does the job. I’ve only ever used with along with --trace so not sure what it looks like with anything else (probably much the same).

defmodule MyApp.Test.Formatter do
  @moduledoc """
  Adds test failures to the bottom of the test output.

  This is mostly only useful when run with `--trace`.

  Usage:

     $ mix test --trace --formatter MyApp.Test.Formatter
  """

  use GenServer

  @format_cols 80

  @doc false
  def init(opts) do
    {:ok, pid} = GenServer.start_link(ExUnit.CLIFormatter, opts)
    {:ok, %{cli_formatter: pid, failures: []}}
  end

  def handle_cast({:test_finished, %{state: {:failed, errors}} = test} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    counter = length(state.failures) + 1

    error =
      ExUnit.Formatter.format_test_failure(test, errors, counter, @format_cols, &formatter/2)

    state = %{state | failures: [error | state.failures]}

    {:noreply, state}
  end

  def handle_cast({:suite_finished, _} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    if Enum.any?(state.failures) do
      failures =
        state.failures
        |> Enum.reverse()
        |> Enum.join("\n")

      IO.puts("""

      #{String.duplicate("=", @format_cols)}

        Failures:

      #{failures}

      #{String.duplicate("=", @format_cols)}
      """)
    end

    {:noreply, state}
  end

  def handle_cast(event, state) do
    GenServer.cast(state.cli_formatter, event)

    {:noreply, state}
  end

  defp formatter(_key, value), do: value
end
5 Likes
6

FWIW, in case this is useful for anyone in the future, here is our test: target in our Makefile:

$ cat Makefile
...
test:
         ...
         time mix test --cover --trace --slowest 4 --timeout 20 --warnings-as-errors || \
                 (mix test --failed; exit 100)
...

If mix test fails, it runs mix test --failed, and returns a non-zero code (in case the failing tests were flakey or the failure was caused by --cover or --warnings-as-errors:

We run make test on our dev machines and in our CI/CD pipelines, and this has been a time saver for us.

7

I adapted this script a little to make it project-agnostic. It now fetches the manifest based on the Mix.Project.manifest_path()

# list_failed_tests.exs
manifest_path = Mix.Project.manifest_path() |> Path.join(".mix_test_failures")

if not File.exists?(manifest_path) do
  IO.puts("Can't find manifest file at #{manifest_path}")
  System.stop()
  Process.sleep(:infinity)
end

manifest_path
|> File.read!()
|> :erlang.binary_to_term()
|> elem(1)
|> Enum.group_by(fn {_, file} -> file end, fn {{_mod, name}, _file} -> name end)
|> Enum.each(fn {file, tests} ->
  IO.puts([
    IO.ANSI.red(),
    "Failures in #{file}:\n",
    Enum.map_join(tests, "\n", &"  * #{&1}"),
    "\n",
    IO.ANSI.reset()
  ])
end)

You can run it with MIX_ENV=test mix run list_failed_tests.exs

3 Likes
8

Hey thank you,

I bookmarked it the check out next week !

9

Hi there! I’ve same issue, that’s why I created simple library GitHub - balance-platform/ex_unit_summary: "Extension" for ExUnit

Feel free to use :slight_smile:

2 Likes