Assertions - helpful assertions to help you write better tests

assertions
#1

Introducing assertions, the library that helps you write really great test assertions!

GitHub: https://github.com/devonestes/assertions
Hex.pm: https://hex.pm/packages/assertions
HexDocs: https://hexdocs.pm/assertions/Assertions.html

This library aims to wrap up common types of assertions that many applications come across in a nice, reusable, composable fashion, and with really exceptional error messages.

Here’s just one example of one use case - comparing a list of structs for equality.

Imagine you have a Phoenix app, and you need to test that the response from a function is basically equal to some other list of structs that you created earlier. We can’t compare them directly because the order of the structs in the list is not guaranteed, so assert list1 == list2 won’t work. Plus, the structs in our list might have differing assocations preloaded or not, meaning we also can’t reliably do assert user1 == hd(list). So, to test this reliably, we’ll need to do something like this:

defmodule UsersTest do
  use ExUnit.Case, async: true

  describe "update_all/2" do
    test "updates the given users in the database and returns those updated users" do
      alice = Factory.insert(:user, name: "Alice")
      bob = Factory.insert(:user, name: "Bob")

      updated_names = 
        [{alice, %{name: "Alice A."}, {bob, %{name: "Bob B."}}}]
        |> Users.update_all()
        |> Enum.map(& &1.name)

      all_user_names =
        User
        |> Repo.all()
        |> Enum.map(& &1.name)

      Enum.each(["Alice A.", "Bob B."], fn name ->
        assert name in updated_names
        assert name in all_user_names
      end
    end
  end
end

But that leaves a lot to be desired. There’s extra code to pull out just the names from the response and we’re obscuring the function that we’re actually testing here. The call to Users.update_all/1 is kind of buried in there.

Let’s see how we might do this with assertions:

defmodule UsersTest do
  use ExUnit.Case, async: true

  describe "update_all/2" do
    test "updates the given users in the database and returns those updated users" do
      alice = Factory.insert(:user, name: "Alice")
      bob = Factory.insert(:user, name: "Bob")

      result = Users.update_all([{alice, %{name: "Alice A."}, {bob, %{name: "Bob B."}}}

      result
        |> Enum.map(& &1.name)
        |> assert_lists_equal(["Alice A.", "Bob B."])

      assert_lists_equal(result, Users.list_all(), &structs_equal?(&1, &2, [:name]))
    end
  end
end

It’s shorter, easier to change, and much more clear about the function being tested in this test. It also gives us really wonderful error messages, which you can see more examples of in the README on GitHub.

There are many more assertions there, and if there are any other common ones that folks uses often in their projects I’d be happy to add them. I want this to be the one library that has all the common helper functions that anyone needs to write really great tests!

9 Likes

#2

This looks very useful! I think it makes sense to use this as an additional base library and then write/name your own assertions on top of it. For example I use this function extensively in my tests:

  @doc """
  Helper for checking that for two structs, or two lists of structs have the
  same id keys
  """
  def assert_ids_match(list1, list2) when is_list(list1) and is_list(list2) do
    list1_ids =
      list1
      |> Enum.map(fn
        %{id: id} -> id
        nil -> nil
      end)
      |> Enum.sort()

    list2_ids =
      list2
      |> Enum.map(fn
        %{id: id} -> id
        nil -> nil
      end)
      |> Enum.sort()

    assert list1_ids == list2_ids
  end

  def assert_ids_match(%{id: id1}, %{id: id2}) do
    assert id1 == id2
  end

And with your assertions library it would look something more like:

import Assertions

def assert_ids_match(left, right) when is_list(left) and is_list(right) do
  assert_lists_equal(left, right, &assert_ids_match/2)
end

def assert_ids_match(%{id: left_id}, %{id: right_id}) do
  assert left_id == right_id
end

I do have a few quibbles with the naming of a few checks. For example I think that assert!/1 would be better named as assert_true or assert_equals_true to make it more obvious that it is checking that the value is exactly equal true (and not just truth). Although a simple assert expression == true is what I would probably still use.

But what I would really like would be a solution that would let you write a test like (where pin operator is used in the same fashion as ecto queries):

result = some_function(data)
assert_pin_match %{id: ^data.id, name: ^original.name} = result

(I talk about this more at Can you pin a temporary variable while asserting a pattern match)

1 Like

#3

For the pin thing, I get around that by doing:

result = some_function(data)
expected = %{id: data.id, name: orginal.name}
assert_maps_equal(result, expected, Map.keys(expected))

Also, one thing to note is that the function you pass to assert_lists_equal should return true or false, not raise an exception (which is what assert does under the hood), so you’d actually want:

import Assertions

assert_lists_equal(left, right, &ids_match?/2)

def ids_match?(%{id: id}, %{id: id}), do: true
def ids_match?(_, _), do: false

This is because the “shrinking” that happens to give the really great error messages relies on being able to compare every element in one list to every element in the other, and if you’re asserting then it’ll fail if there is any element in one list that isn’t equal to any other element in the other.

However, what I do is have functions like structs_equal? or maps_equal? that take a set of keys to use to determine if the two elements are equal in this scenario, like this:

import Assertions

assert_lists_equal(left, right, &structs_equal?(&1, &2, [:id]))

def structs_equal?(left, right, keys) do
  Enum.all(keys, fn key -> Map.get(left, key) == Map.get(right, key) end)
end

But that’s totally the intention behind all these assertions - to give you the basics that then allows you to use your own comparison functions to determine equality for the situation you’re in. == just doesn’t always cut it, ya know :wink:

2 Likes

#4

I wonder if you should use assert lists_equal(...) instead? From Elixir v1.7 or v1.8 we break the arguments apart in the output so there is a chance the default output is quite good without creating new assertions but only with new auxiliary functions. How do the error outputs compare?

2 Likes

#5

That looks pretty neat. Can you link to where these are documented? I googled, but I couldn’t find anything.

EDIT: Oh wait, I guess I misinterpreted this… You mean you could write a function to do the equal and then use inside the assert, not that there are predefined functions for this in ExUnit or somewhere, right? :smile:

2 Likes

#6

The improvements in 1.8 are very helpful, but it doesn’t do the shrinking that I do in assertions, and it also doesn’t expand all variables in the argument to assert. For example, a common way to test unordered list equality is something like this:

list = [1, 2, 3]
assert Enum.all?([2, 1, 4], & &1 in list)

That gives us this failure:

  1) test example (AssertionsTest)
     test/assertions_test.exs:8
     Expected truthy, got false
     code: assert Enum.all?([2, 1, 4], &(&1 in list))
     arguments:

         # 1
         [2, 1, 4]

         # 2
         #Function<6.120201396/1 in AssertionsTest."test example"/1>

     stacktrace:
       test/assertions_test.exs:10: (test)

So we can see the one list, but we can’t see the elements in the other list that we’re checking for unordered equality. And then there’s also the problem of that not actually checking equality, which many people think it does :wink:

But if we use assert_lists_equal then we write the test like this:

list = [1, 2, 3]
assert_lists_equal(list, [2, 1, 4])

Which gives us this output:

  1) test example (AssertionsTest)
     test/assertions_test.exs:8
     Comparison of each element failed!
     code:  assert_lists_equal(list, [2, 1, 4])
     arguments:

         # 1
         [1, 2, 3]

         # 2
         [2, 1, 4]

     left:  [3]
     right: [4]
     stacktrace:
       test/assertions_test.exs:10: (test)

So there we can see the differing elements between the two lists in left: and right: , which I find really helpful. Since the standard diffing that we do with something like list1 == list2 assumes that order matters, it doesn’t work when comparing lists where the order doesn’t matter (which is a great deal of the time). I thought the best way to show the diff would be to remove the common elements.

I also do this shrinking when comparing structs/maps with assert_maps_equal/3 - it will just show you what’s different instead of showing you the whole big map, and you still get the colored diff if the maps are nested.

5 Likes

#7

Beautiful.

4 Likes

#8

Looks like a helpful library to, definitely going to check it out.

You might have picked a different name though, because right now, googling for “elixir assertions” will only give you results for ExUnit :wink:

1 Like

#9

I’ve added a new feature in the newest release (0.13.0) that might be of interest to folks.

Now, if you try and do:

assert! nil > 0

your assertion will fail even though that assertion would pass if using assert. If you try and use nil with any of the ordinal comparison operators (>, <, >=, <=) it is assumed to be a bug. This is what the failure message looks like:

  1) test assert!/1 fails when using nil (Assertions.FailureExamples)
     test/failure_examples.exs:24
     `nil` is not allowed as an argument to `>` when using `assert!`
     code:  assert! nil > 0
     left:  nil
     right: 0
     stacktrace:
       test/failure_examples.exs:25: (test)

This is also the behavior for refute! as well.

I’m also planning on implementing further safety checks like this with ordinal comparison operators, such as disallowing using them to compare date/time/datetime structs as that will also not work as folks expect and should be considered a bug in your test.

I may also go a step further and only allow use of the ordinal comparison operators for a restricted set of “similar” types, so you can only use them to compare two strings, or two atoms, or two floats or integers, and not compare (for example) a string to an integer, or a Decimal struct with an integer. I don’t know of any use case where a developer would intentionally want to use > to assert that a string is greater than an integer, but if someone can tell me one I’d love to hear it!

2 Likes

#10

Sorting a list of unknown contents.

1 Like

#11

That might have been unclear - I mean to assert in a test that a string is greater than in integer. Your example wouldn’t be used in assert! - that’s internal to Enum.sort or whatever you’re using for sorting. I’m talking about a bare

assert! "10" > 10

assertion.

1 Like

#12

In the implementation of a sorted set I might want to test if succeeding items are in the correct decreasing (or increasing) order.

1 Like