Handling multi-newline exception messages in doctests? (Elixir v1.12.0)

Questions / Help
1

Hi Forum,

Has anyone writing doctests found a way to update their examples that produce ArgumentErrors to capture the new multiline format produced in Elixir 1.12.0? I had some trouble updating an error expectation today because of the double newline.

Here’s what succeeded in Elixir 1.11.4:

@doc ~S"""
  Returns `map` with its keys as atoms, if those atoms already exist.

  Raises `ArgumentError` otherwise.

  ## Examples

      iex> atomize_keys!(%{"oh" => "ooh", "noo" => "noooo"})
      ** (ArgumentError) argument error
"""

But updating to Elixir 1.12.0’s reported actual results in this test failure:

Doctest failed: wrong message for ArgumentError
expected:
  "errors were found at the given arguments:\\n\\n  * 1st argument: invalid UTF8 encoding\\n"
actual:
  "errors were found at the given arguments:\n\n  * 1st argument: invalid UTF8 encoding\n"

And if I update the doc tag to interpret escaped characters (i.e., @doc """), the test just fails in the opposite direction:

Doctest failed: wrong message for ArgumentError
expected:
  "errors were found at the given arguments:"
actual:
  "errors were found at the given arguments:\n\n  * 1st argument: invalid UTF8 encoding\n"

…And, yes, escaping the escape fails to improve the situation:

Doctest failed: wrong message for ArgumentError
expected:
  "errors were found at the given arguments:\\n\\n  * 1st argument: invalid UTF8 encoding\\n"
actual:
  "errors were found at the given arguments:\n\n  * 1st argument: invalid UTF8 encoding\n"

Me @ this point: tyra was rooting for escaped backslash

I implemented a new exception as a workaround for my use case, but it feels like a circumstance that does deserve a more generally flexible fix. I mean, what if a library developer does want to use an example of a multiline exception with double newlines in one of their doctests? Clearly, there are cases now where at least ArgumentErrors produce such exceptions, and those cases shouldn’t fail in the test runs. At least, it doesn’t feel like they should ._.

3 Likes
2

Doctests do not support multi-line exceptions because it is hard for it to know when the exception is over and when you have a new paragraph (as we typically avoid relying on indentation). However, we definitely need to come up with a mechanism to do so… or at least for a subtext match. Please open up an issue!

6 Likes
3

workaround for me was to use assert_raise without any output check, not ideal to use asserts in doctests but does the job: iex> assert_raise ArgumentError, fn -> atomize_keys!(%{"oh" => "ooh", "noo" => "noooo"}) end

4

Doctests now (maybe as of 1.14) support multiline exceptions as documented here:

https://hexdocs.pm/ex_unit/1.15.5/ExUnit.DocTest.html#module-exceptions

However, as far as I can tell it isn’t currently possible to add a doctest that’ll handle hd([]) because the generated exception has a blank line:

** (ArgumentError) errors were found at the given arguments:

  * 1st argument: not a nonempty list

(not including the somewhat verbose assert_raise workaround that @hypno mentioned above)

Here’s a quick repro repository: GitHub - axelson/multiline_doctest_repro (run mix test to see the error for yourself)

So now that we have multi-line exception support in doctests, could we also consider some means to indicate a subtext match on the exception?

Or maybe we could support multi-line exceptions by collapsing blank lines into a single line? That would make this doctest work:

iex> hd([])
** (ArgumentError) errors were found at the given arguments:
  * 1st argument: not a nonempty list
2 Likes
5

I thought several times about making it a prefix match on the error message, but then it means including less information in the example. :frowning: Also, if we remove the empty lines, we will definitely receive pull requests adding the lines back. I am not quite sure how to address this.

1 Like
6

Hmm, maybe a pragmatic solution is in a doctest that is checking for an exception, only match the first line. So this would be a valid passing doctest:

iex> hd([])
** (ArgumentError) errors were found at the given arguments:

Although it definitely still feels a bit weird.

1 Like
7

I’m not using doctests often so I’m not super familiar with them. But shouldn’t it be possible to use the actual exception message to check if that full message appears in the doctest?

Now that I think about it. That’s probably hard because of the way doctests are parsed?

8

i use doctests all the time now. especially for advent of code. Not having to define tests in different file is so nice. Not having to even open repl. Just mix.watch --only etc for a tagged doctest. (for ex mix test.watch --only day:2024.d12.p1)

I ususally have to define multiline input in another module. And i have to be super careful of not having space between the tests, if i have one doc test and then a new line … the other tests are not run

for ex this passes green, which should not

  @doc ~S"""
  ## Examples

      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.test_input_1())
      0

      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.test_input_2())
      772
      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.test_input())
      1930
      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.input())
      0

  """
  def solution(_input), do: 0

if i remove the space, things start properly failing again

  @doc ~S"""
  ## Examples

      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.test_input_1())
      140
      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.test_input_2())
      772
      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.test_input())
      1930
      iex> AOC2024.Day12.Part1.Solution.solution(AOC2024.Day12.Input.input())
      0

  """
  def solution(_input), do: 0

so its fragile and error prone. For this reason i should really not depend on doctests for production code … yet …