Testing for expected failure

Questions / Help
,
1

Hi,

How would one make a test that asserts that the tested method is invalid?

If i for example want to test the following (somewhat contrived) method:

def only_valid_number(number) when is_number number do
  number
end

A simple success test is trivial:

assert 5 == only_valid_number(5)

But what if I want to verify that:

assert ‘got stacktrace’ == only_valid_number(“5”)

Where I don’t really care why, just that any attempts to use it in this way will fail.

2

assert_raise…

https://hexdocs.pm/ex_unit/ExUnit.Assertions.html#assert_raise/2

4 Likes
3

As I do guess this is as close to the right answer I will get it is (unfortunately) not what I’m after. The difference being that, as far as I understand, there is no way to not specify which exception specifically to expect.

I would like to find a way to just throw stuff it should not except at the function and then rest assured that I did crash in a controlled way.

As I see it if you specify an expected exception a bit of the Erlang magic is gone when you have to predict the exact behavior and thus only cover one of all the (infinite) possible errors.

4

Perhaps I should rephrase my vision as:

assert failure == only_valid_number(“5”)

where failure is anything that does not return a valid response

5

You could use flunk instead to check that the execution does not reach a certain point:

try do
  only_valid_number("x")
  flunk("this should not have happened")
catch
  _, _ -> :ok 
end
6

To bad you have to wrap it in stuff but this is it.

7

My suggestion was not entirely right. You could write a expect_failure macro like this:

defmodule ExpectFailure do
  defmacro expect_failure(ast) do
    quote do
      try do
        unquote(ast)
        flunk("This should has crashed: " <> unquote(Macro.to_string(ast)))
      rescue
        e in ExUnit.AssertionError ->
          raise e

        _ ->
          :ok
      end
    end
  end
end

defmodule AsdfTest do
  use ExUnit.Case
  doctest Asdf
  import ExpectFailure

  def foo(x) when is_number(x) do
    x + 1
  end

  test "expect_failure catches errors" do
    expect_failure(foo("a"))
  end

  test "expect_failure should crash" do
    assert_raise ExUnit.AssertionError, fn ->
      expect_failure(foo(1))
    end
  end
end
1 Like
8

This was a good idea! I’ve added it to the new version (0.15.0) and just published it to hex: https://hex.pm/packages/assertions
https://hexdocs.pm/assertions/0.15.0/Assertions.html#assert_raise/1

2 Likes