Handling multi-newline exception messages in doctests? (Elixir v1.12.0)

Questions / Help
1

Hi Forum,

Has anyone writing doctests found a way to update their examples that produce ArgumentErrors to capture the new multiline format produced in Elixir 1.12.0? I had some trouble updating an error expectation today because of the double newline.

Here’s what succeeded in Elixir 1.11.4:

@doc ~S"""
  Returns `map` with its keys as atoms, if those atoms already exist.

  Raises `ArgumentError` otherwise.

  ## Examples

      iex> atomize_keys!(%{"oh" => "ooh", "noo" => "noooo"})
      ** (ArgumentError) argument error
"""

But updating to Elixir 1.12.0’s reported actual results in this test failure:

Doctest failed: wrong message for ArgumentError
expected:
  "errors were found at the given arguments:\\n\\n  * 1st argument: invalid UTF8 encoding\\n"
actual:
  "errors were found at the given arguments:\n\n  * 1st argument: invalid UTF8 encoding\n"

And if I update the doc tag to interpret escaped characters (i.e., @doc """), the test just fails in the opposite direction:

Doctest failed: wrong message for ArgumentError
expected:
  "errors were found at the given arguments:"
actual:
  "errors were found at the given arguments:\n\n  * 1st argument: invalid UTF8 encoding\n"

…And, yes, escaping the escape fails to improve the situation:

Doctest failed: wrong message for ArgumentError
expected:
  "errors were found at the given arguments:\\n\\n  * 1st argument: invalid UTF8 encoding\\n"
actual:
  "errors were found at the given arguments:\n\n  * 1st argument: invalid UTF8 encoding\n"

Me @ this point: tyra was rooting for escaped backslash

I implemented a new exception as a workaround for my use case, but it feels like a circumstance that does deserve a more generally flexible fix. I mean, what if a library developer does want to use an example of a multiline exception with double newlines in one of their doctests? Clearly, there are cases now where at least ArgumentErrors produce such exceptions, and those cases shouldn’t fail in the test runs. At least, it doesn’t feel like they should ._.

3 Likes
2

Doctests do not support multi-line exceptions because it is hard for it to know when the exception is over and when you have a new paragraph (as we typically avoid relying on indentation). However, we definitely need to come up with a mechanism to do so… or at least for a subtext match. Please open up an issue!

6 Likes
3

workaround for me was to use assert_raise without any output check, not ideal to use asserts in doctests but does the job: iex> assert_raise ArgumentError, fn -> atomize_keys!(%{"oh" => "ooh", "noo" => "noooo"}) end

4

Doctests now (maybe as of 1.14) support multiline exceptions as documented here:

https://hexdocs.pm/ex_unit/1.15.5/ExUnit.DocTest.html#module-exceptions

However, as far as I can tell it isn’t currently possible to add a doctest that’ll handle hd([]) because the generated exception has a blank line:

** (ArgumentError) errors were found at the given arguments:

  * 1st argument: not a nonempty list

(not including the somewhat verbose assert_raise workaround that @hypno mentioned above)

Here’s a quick repro repository: GitHub - axelson/multiline_doctest_repro (run mix test to see the error for yourself)

So now that we have multi-line exception support in doctests, could we also consider some means to indicate a subtext match on the exception?

Or maybe we could support multi-line exceptions by collapsing blank lines into a single line? That would make this doctest work:

iex> hd([])
** (ArgumentError) errors were found at the given arguments:
  * 1st argument: not a nonempty list
2 Likes
5

I thought several times about making it a prefix match on the error message, but then it means including less information in the example. :frowning: Also, if we remove the empty lines, we will definitely receive pull requests adding the lines back. I am not quite sure how to address this.

1 Like
6

Hmm, maybe a pragmatic solution is in a doctest that is checking for an exception, only match the first line. So this would be a valid passing doctest:

iex> hd([])
** (ArgumentError) errors were found at the given arguments:

Although it definitely still feels a bit weird.

1 Like
7

I’m not using doctests often so I’m not super familiar with them. But shouldn’t it be possible to use the actual exception message to check if that full message appears in the doctest?

Now that I think about it. That’s probably hard because of the way doctests are parsed?