Can you cause ex_unit test suite to fail if there's extraneous output?

Questions / Help
,
1

At work we have a rather large test suite that we’ve built up over multiple years. Many of the tests are veering towards integration tests (e.g. tests of Channels or Controllers) and some of them start extraneous processes during the test.

To help keep the test suite manageable it would be very helpful if there was a way to cause the test suite to fail if any extraneous output was printed during the test suite run, that way we can ensure we’re taking a more disciplined approach to our tests. Examples of output that might be printed during a test suite run are:

  • Mox or Ecto Sandbox errors in processes that linger beyond the end of the originating test
  • Logger or IO.puts statements that haven’t been captured with ExUnit.CaptureLog/IO
  • Ecto DBConnection.Connection errors
2

Hi, Jason :wave:

I recently put together a script for a similar purpose, though we only run it in CI.

Put it in a file named clean_test_warden.exs and run it with:

(mix test --warnings-as-errors; echo "mix test exit code $?") | elixir clean_test_warden.exs

Here is the script.

##
# This script is there to run in CI pieplie and fail it
# if there is any garbage within the test output
#
# Reads the STDIO line by line.
#
# ## Usage example:
#
#    $  (mix test --warnings-as-errors; echo "mix test exit code $?") | elixir clean_test_warden.exs
#
#
# ## Implementation details
#
# The accumulator is a tuple of the final exit code and a flag if it is supposed to analyze the line
#
# initial `exit_code` is `0`
# initial `analyze?` is `false`
#
# Basically it ignores lines until it sees the line that starts with
# either "Including tags:" or "Excluding tags:"
# then it sets `analyze?` to `true`
#
# We only expect to see the following characters:
#
#   `.` - passed test
#   `*` - skipped test
#   `F` - failed test
#
#   (and some newlines)
#
# When `analyze?` is `true` it disregards the next `\n` (newline)
# on the following line it removes all `.`, `*` and `F`
# and validates that in the end it got nothing but a trailing newline.
# 
# We expect the test suite run to complete followed by a line
# that start with `"Finished"`
#
# Finally in order to preserve the original exit code of `mix test` we expect it to be passed along with a string
#   `mix test exit code <EXIT_CODE>`
#
# (for example if we run `mix test` with a flag `--warnings-as-errors`
# it will return non-zero exit with the intention cause the pipeline fail)

input_stream = IO.stream(:stdio, :line)

{exit_code, _analyze?} =
  Enum.reduce(input_stream, {0, false}, fn
    "Finished" <> _ = line, {exit_code, _analyze?} ->
      IO.write(line)
      {exit_code, false}

    line, {exit_code, _analyze?} = acc when exit_code != 0 ->
      IO.write(line)
      acc

    "Including tags:" <> _ = line, {exit_code, _analyze?} ->
      IO.write(line)
      {exit_code, true}

    "Excluding tags:" <> _ = line, {exit_code, _analyze?} ->
      IO.write(line)
      {exit_code, true}

    "mix test exit code " <> original_exit_code, {exit_code, analyze?} ->
      {original_exit_code, _} = Integer.parse(original_exit_code)

      if original_exit_code != 0,
        do: {original_exit_code, analyze?},
        else: {exit_code, analyze?}

    line, {_exit_code, analyze?} = acc ->
      if analyze? and line != "\n" and String.replace(line, ~r/[\.\*F]/, "") != "\n" do
        IO.write(line)
        {1, analyze?}
      else
        IO.write(line)
        acc
      end
  end)

if exit_code != 0 do
  IO.puts(:stderr, "\n\nTest output was not clean\n\n")
end

exit({:shutdown, exit_code})

It’s not ideal… but it gets the work done and doesn’t let the tests with garbage in the output to be merged to the main branch :slightly_smiling_face:

2 Likes
3

I wonder if a custom test formatter can help enable this. :thinking:

For example something like GitHub - joshwlewis/tapex: TAP (Test Anything Protocol) formatter for Elixir's ExUnit? Then you can feed the output to a script and assert it strictly adheres to a format.

Definitely hacky though.

1 Like
4

Oh, wow! I didn’t know about ExUnit formatters! Thank you!