How to get better diff comparison for strings with newlines?

It sounds like something you could do with the ExUnit.Formatter module.

format_test_failure(test, failures, counter, width, formatter)
Receives a test and formats its failures.

Examples

iex> failure = {:error, catch_error(raise "oops"), _stacktrace = []}
iex> formatter_cb = fn _key, value -> value end
iex> test = %ExUnit.Test{name: :"it works", module: MyTest, tags: %{file: "file.ex", line: 7}}
iex> format_test_failure(test, [failure], 1, 80, formatter_cb)
"  1) it works (MyTest)\n     file.ex:7\n     ** (RuntimeError) oops\n"

source: docs for format_test_failure/5

The problem I run into here is that format_test_failure seems to escape \n as \\n. AFAIK (and I would love to be proved wrong) I think you have to do the formatting yourself.

This is a cobbled together version from a formatter I have. I was hooking into the standard formatter and using format_test_failure so I had to remove that which of course removes all the meta data like test name and line number (though it’s easily added as all of that info is available between error and meta). All this has is left and right but it does work if you want to use it as a starting point!

Again, hopefully I’m wrong and there is an easier way.

defmodule Formatter do
  use GenServer

  def init(_opts) do
    {:ok, %{failures: []}}
  end

  def handle_cast({:test_finished, %{state: {:failed, errors}} = _test} = _event, state) do
    error =
      errors
      |> Enum.map(fn {:error, error, _meta} ->
        left =
          String.split(error.left, "\n")
          |> Enum.map(& "      " <> &1)
          |> Enum.join("\n")

        right =
          String.split(error.right, "\n")
          |> Enum.map(&"       " <> &1)
          |> Enum.join("\n")

        "left: \"\"\"\n" <> left <> "\"\"\"\n\nright: \"\"\"\n" <> right <> "\"\"\""
      end)
      |> Enum.join("\n")

    state = %{state | failures: [error | state.failures]}

    {:noreply, state}
  end

 def handle_cast({:suite_finished, _} = _event, state) do
   if Enum.any?(state.failures) do
     failures =
       state.failures
       |> Enum.reverse()
       |> Enum.join("\n")

     IO.puts("""

       Failures:

     #{failures}
     """)
   end

   {:noreply, state}
 end

  def handle_cast(_event, state) do
    {:noreply, state}
  end
end

ExUnit.start(formatters: [Formatter])

defmodule TestIt do
  use ExUnit.Case

  test "thing" do
    assert """
    line 1
    line 2
    line 3
    """ == """
    line oops
    line 2
    line 3
    """
  end
end

I’ve been doing similar things at work for showing meaningful diffs between 2 JSON files. I had to normalize both: sort keys (json keys don’t have order), pretty print, and only then diff.

See these links for inspiration:

1. Semantic Diff for SQL
2. pg_query fingerprint
3. difftastic for SQL

Thanks for the suggestions. I was definitely in the headspace of thinking I’d have to write a custom assertion. I forgot ex_unit even had formatters, so that will be a useful area to explore.