Assertions - helpful assertions to help you write better tests

This looks very useful! I think it makes sense to use this as an additional base library and then write/name your own assertions on top of it. For example I use this function extensively in my tests:

  @doc """
  Helper for checking that for two structs, or two lists of structs have the
  same id keys
  """
  def assert_ids_match(list1, list2) when is_list(list1) and is_list(list2) do
    list1_ids =
      list1
      |> Enum.map(fn
        %{id: id} -> id
        nil -> nil
      end)
      |> Enum.sort()

    list2_ids =
      list2
      |> Enum.map(fn
        %{id: id} -> id
        nil -> nil
      end)
      |> Enum.sort()

    assert list1_ids == list2_ids
  end

  def assert_ids_match(%{id: id1}, %{id: id2}) do
    assert id1 == id2
  end

And with your assertions library it would look something more like:

import Assertions

def assert_ids_match(left, right) when is_list(left) and is_list(right) do
  assert_lists_equal(left, right, &assert_ids_match/2)
end

def assert_ids_match(%{id: left_id}, %{id: right_id}) do
  assert left_id == right_id
end

I do have a few quibbles with the naming of a few checks. For example I think that assert!/1 would be better named as assert_true or assert_equals_true to make it more obvious that it is checking that the value is exactly equal true (and not just truth). Although a simple assert expression == true is what I would probably still use.

But what I would really like would be a solution that would let you write a test like (where pin operator is used in the same fashion as ecto queries):

result = some_function(data)
assert_pin_match %{id: ^data.id, name: ^original.name} = result

(I talk about this more at Can you pin a temporary variable while asserting a pattern match)

For the pin thing, I get around that by doing:

result = some_function(data)
expected = %{id: data.id, name: orginal.name}
assert_maps_equal(result, expected, Map.keys(expected))

Also, one thing to note is that the function you pass to assert_lists_equal should return true or false, not raise an exception (which is what assert does under the hood), so you’d actually want:

import Assertions

assert_lists_equal(left, right, &ids_match?/2)

def ids_match?(%{id: id}, %{id: id}), do: true
def ids_match?(_, _), do: false

This is because the “shrinking” that happens to give the really great error messages relies on being able to compare every element in one list to every element in the other, and if you’re asserting then it’ll fail if there is any element in one list that isn’t equal to any other element in the other.

However, what I do is have functions like structs_equal? or maps_equal? that take a set of keys to use to determine if the two elements are equal in this scenario, like this:

import Assertions

assert_lists_equal(left, right, &structs_equal?(&1, &2, [:id]))

def structs_equal?(left, right, keys) do
  Enum.all(keys, fn key -> Map.get(left, key) == Map.get(right, key) end)
end

But that’s totally the intention behind all these assertions - to give you the basics that then allows you to use your own comparison functions to determine equality for the situation you’re in. == just doesn’t always cut it, ya know

I wonder if you should use assert lists_equal(...) instead? From Elixir v1.7 or v1.8 we break the arguments apart in the output so there is a chance the default output is quite good without creating new assertions but only with new auxiliary functions. How do the error outputs compare?

That looks pretty neat. Can you link to where these are documented? I googled, but I couldn’t find anything.

EDIT: Oh wait, I guess I misinterpreted this… You mean you could write a function to do the equal and then use inside the assert, not that there are predefined functions for this in ExUnit or somewhere, right?

The improvements in 1.8 are very helpful, but it doesn’t do the shrinking that I do in assertions, and it also doesn’t expand all variables in the argument to assert. For example, a common way to test unordered list equality is something like this:

list = [1, 2, 3]
assert Enum.all?([2, 1, 4], & &1 in list)

That gives us this failure:

  1) test example (AssertionsTest)
     test/assertions_test.exs:8
     Expected truthy, got false
     code: assert Enum.all?([2, 1, 4], &(&1 in list))
     arguments:

         # 1
         [2, 1, 4]

         # 2
         #Function<6.120201396/1 in AssertionsTest."test example"/1>

     stacktrace:
       test/assertions_test.exs:10: (test)

So we can see the one list, but we can’t see the elements in the other list that we’re checking for unordered equality. And then there’s also the problem of that not actually checking equality, which many people think it does

But if we use assert_lists_equal then we write the test like this:

list = [1, 2, 3]
assert_lists_equal(list, [2, 1, 4])

Which gives us this output:

  1) test example (AssertionsTest)
     test/assertions_test.exs:8
     Comparison of each element failed!
     code:  assert_lists_equal(list, [2, 1, 4])
     arguments:

         # 1
         [1, 2, 3]

         # 2
         [2, 1, 4]

     left:  [3]
     right: [4]
     stacktrace:
       test/assertions_test.exs:10: (test)

So there we can see the differing elements between the two lists in left: and right: , which I find really helpful. Since the standard diffing that we do with something like list1 == list2 assumes that order matters, it doesn’t work when comparing lists where the order doesn’t matter (which is a great deal of the time). I thought the best way to show the diff would be to remove the common elements.

I also do this shrinking when comparing structs/maps with assert_maps_equal/3 - it will just show you what’s different instead of showing you the whole big map, and you still get the colored diff if the maps are nested.

Beautiful.

Looks like a helpful library to, definitely going to check it out.

You might have picked a different name though, because right now, googling for “elixir assertions” will only give you results for ExUnit

I’ve added a new feature in the newest release (0.13.0) that might be of interest to folks.

Now, if you try and do:

assert! nil > 0

your assertion will fail even though that assertion would pass if using assert. If you try and use nil with any of the ordinal comparison operators (>, <, >=, <=) it is assumed to be a bug. This is what the failure message looks like:

  1) test assert!/1 fails when using nil (Assertions.FailureExamples)
     test/failure_examples.exs:24
     `nil` is not allowed as an argument to `>` when using `assert!`
     code:  assert! nil > 0
     left:  nil
     right: 0
     stacktrace:
       test/failure_examples.exs:25: (test)

This is also the behavior for refute! as well.

I’m also planning on implementing further safety checks like this with ordinal comparison operators, such as disallowing using them to compare date/time/datetime structs as that will also not work as folks expect and should be considered a bug in your test.

I may also go a step further and only allow use of the ordinal comparison operators for a restricted set of “similar” types, so you can only use them to compare two strings, or two atoms, or two floats or integers, and not compare (for example) a string to an integer, or a Decimal struct with an integer. I don’t know of any use case where a developer would intentionally want to use > to assert that a string is greater than an integer, but if someone can tell me one I’d love to hear it!

Sorting a list of unknown contents.

That might have been unclear - I mean to assert in a test that a string is greater than in integer. Your example wouldn’t be used in assert! - that’s internal to Enum.sort or whatever you’re using for sorting. I’m talking about a bare

assert! "10" > 10

assertion.